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Executive Summary 


This report provides a description of a study conducted by WestEd on behalf of the Bill and 
Melinda Gates Foundation to examine how the affordances of the technologies in the blended 
learning environments of three charter schools impacted teaching and learning activities. A 
particular focus of the work was to examine whether the blended learning environments provided 
enhanced access to and more diverse data sources for teachers and students from which to make 
educational decisions. 


The research was conducted in three small charter schools in Arizona, serving populations of 
challenged middle and high school students. The schools were infused with technologies that 
support in-person and distant educational experiences, using a hybrid model. Teachers and 
administrators were well trained in the use of the technologies and how they could be used to 
enhance instructional and administrative decision making. The school administrators provided 
for ongoing professional development, technical assistance, and other learning opportunities to 
assist the teachers to effectively integrate the technologies into the teaching and learning process. 
Findings from the research are consistent with results from related studies on blended learning 
and recommendations for the effective use of data. 


The school administrators provided strong leadership and a clear vision for the use of the 
technologies and data to enhance teaching and learning. They provided the needed resources to 
make possible the effective use of both the technologies and data in the form of ongoing 
professional learning opportunities. They created and enculturated the use of technology and data 
through data teams, data coaches, and dedicated planning time. They involved not only the 
teachers but also the students as active participants in the teaching and learning process. 
Technologies were used to enhance the process both within the school buildings and virtually. 
Students had ready access to a diversity of technological solutions at school and outside of 
school. The blended learning environments provided for any time, any where access to 
educational activities for the students. This was an essential component given the many 
challenges faced by the students. Students could access instructional and assessment materials 
via mobile devices at their convenience. The hybrid model was not reliant on in-person 
attendance in class, although it was strongly recommended. The environments allowed for 
customization to personalized learning goals and needs with flexible pathways to learning for 
each student. 


The research used a purposive sample of three schools, leaving open the question of whether the 
findings can generalize to other similar charter schools as well as to more traditional schools. Not 
all schools are steeped in technology. Not all schools have strong leadership with targeted 
visions on how technology and data can be used to meet the needs of all students. Not all schools 
can provide the kinds of resources to build and sustain the human capacity of the educational 
staff to use the technologies and data. 


Yet even with these resources, the schools faced challenges. Despite the plethora of 
technological solutions, the data from them were siloed, meaning that there was little 


interoperability and connectedness. There was no single data repository. The lack of 
interoperability created problems in terms of ease of access and the linking of diverse sources of 
data. Bandwidth was an issue. The heavy reliance on connectivity within the school buildings 
meant that there must be substantial bandwidth to accommodate all the technologies. 
Connectivity beyond the schools also was an issue. Most, if not all students, had mobile devices 
of their own. However, they might not have data plans, creating the need to find WiFi enabled 
sites from which to do their work virtually. This was not always the case. Not all students had 
WiFi at their home situations. 


In terms of the teachers, the blended learning process provided both opportunities and 
challenges. Teachers had access to a multitude of resources. Teachers needed to be creative and 
thoughtful about how they used the resources. There was a continuous learning process for them 
as well as their students. For some, it took them out of their comfort zone by changing the 
structure of the traditional classroom. Students were more engaged and active. Students were 
also engaged any time and any where, which meant that teachers were sometimes working 24/7 
and certainly beyond the confines of the school. The labor intensiveness of the teaching and 
learning process can take its toll on the staff, through turnover and mobility. 


There is no doubt that education, whether in traditional or blended environments, is a challenging 
enterprise. That said, the affordances of the technologies in the three schools in this project 
provided opportunities in the teaching and learning process that most likely would have not been 
possible if they were in more traditional settings. The teachers were able to address the needs of 
particular students through various media and diverse learning experiences. The ability to move 
the educational process beyond the school walls seemingly was an essential component. The 
ability to engage the students by providing flexible and customizable learning activities also was 
a key component. The technologies made possible such personalization. Other characteristics of 
these schools are, however, generalizable to traditional environments — strong leadership, an 
explicit vision for the use of technology and data, the engagement of students in the teaching and 
learning process, the enculturation of data use through data teams and data coaches, and the 
provision of professional learning opportunities. The three schools were fortunate to have these 
important components, yet there are other challenges to overcome. Education is difficult and 
complex at best. The blended learning process helped the teachers in these schools better meet 
the needs of their students. That is progress. 


Introduction! 


The purpose of this paper is to explore how the affordances of blended learning environments 
provide data for educational decision making to administrators, teachers, and students that might 
not be otherwise available. Because blended learning environments have significant technology 
infusion, those applications are likely to yield a plethora of data that can assist educators to 
inform their practice in formats and real-time feedback loops not possible in more traditional 
educational settings, while also engaging students in the examination of their own data. The 
blended environments may alter the structure and functioning of classrooms by changing how 
instructional activities are designed and delivered and by making the role of students more active 
participants in the teaching and learning process. 


Blended and personalized learning have become topics of emphasis for policymakers. 
Personalized learning is seen as the customization of educational and instructional activities to 
address individual students’ needs. Yet as Cavanagh (2014) notes, there still is no common 
definition despite the increasing discourse around personalized learning. A recent event, Time to 
Act: Making Data Work for Students, held by the Data Quality Campaign (2016) brought 
together policymakers, elected government officials, educators, civic leaders, and other 
interested stakeholders to discuss the role of data in improving education. At the heart of the 
discussion was personalized learning. Participants noted the importance of personalized learning 
environments, its technology, and the resulting data, to customize instruction and education to 
meet the needs of all students (Guidera, 2016; Holiday, 2016; Messer & Polis, 2016; Perdue, 
2016). The speakers described the power of personalized learning: the provision for real-time 
data; the empowerment of teachers and students; immediate course corrections; the use of data to 
individualize for every student; and connectivity to reach all students regardless of 
circumstances. Guidera (2016) laid out several important points that pertain to the model of 
learning. First, education must make students the central focus. Data should be used to address 
the individual needs of each student. Second, the technology is not enough; the focus must be on 
the people, not the data systems or the technology. Third, education must move away from a 
compliance model and instead use data to improve the teaching and learning process. Fourth, 
there needs to be a culture of continuous improvement with the provision for the right data at the 
right level of granularity for the right solutions. Personalized and blended learning environments 
both have the capacity to address student needs if properly implemented. 


Blended learning and personalized learning are often used interchangeably, although there are 
differences. Blended learning is a somewhat anomalous concept with differing definitions, 
differing terminology, and varying components. This report focuses on the term, blended 
learning because it melds the use of technology and data sources into the structure and 
functioning of the school and the classrooms. It is a hybrid model that combines the unique 
affordances of the technologies, yet provides the face-to-face interactions of brick and mortar 
schools and classrooms. Because of the technology infusion in blended learning, these 
environments are thought to provide more opportunities for new, diverse, and real-time data 
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invaluable information about the Galileo system. 
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collection, not possible without the technology. As such, these environments contain some of the 
customizable components of personalized learning (Bailey et al., 2015). 


Research in the areas of blended and personalized learning has begun to emerge. In particular, 
the research reported here is informed by the work of Pane, Steiner, Baird, and Hamilton (2015) 
conducted at RAND Education. Using a multi-method approach, the RAND research identified 
five domains or topics and 13 attributes of personalized learning that inform the current work, 
particularly in interpreting the results. 
¢ Learner Profiles 
o Personalized goals for students 
o Student data provided and discussed with students 
¢ Personal Learning Path 
o Outside of school learning 
co Flexible/multiple paths for students through content 
o Individual student support 
* Competency-Based Progression 
o Student progress through content based on competency 
o On-demand assessment to demonstrate competency 
¢ Flexible Learning Environment 
o Student grouping 
o Learning space supports model 
o Structure of learning time 
o Extended learning time for students (extended school day or year) 
o Technology available to all students 
¢ College and Career Readiness 
o Developing college/career preparation skills 


RAND and WestEd researchers communicated about common interests so that both studies 
would be mutually informed. The RAND attributes were published after WestEd’s data 
collection was completed thereby helping to inform the coding and analyses of the project’s data 
sources. 


The WestEd report includes a review of the existing literature and is intended to contextualize 
the case studies presented here in the broader landscape of blended learning research. To provide 
the needed context, this review examines the literature in terms of five topics: 
¢ What is blended learning? 
¢ What are the anticipated affordances or advantages of K-12 blended learning 
environments; what are the anticipated challenges; and what are the trends? 
¢ What is the national context that influences blended learning? 
¢ What does research tell us about blended learning implementation, particularly with 
regard to needs for, access to, and use of data to inform instruction? 
¢ What does research tell us about outcomes for students in blended learning 
environments? 


Because the field is still relatively new, rigorous research and evaluation of blended learning’s 
K-12 implementation and outcomes are also relatively few. Most frequently cited research and 
evaluation on these topics are in the form of case studies, although the RAND study linked 


elements of personalized learning to student outcomes (Pane et al., 2015). This review examines 
a broad range of research on blended and personalized learning to provide an overview of what is 
known about K-12 blended or personalized learning environments for students, especially with 
regard to data use for informing instruction and learning. Although blended learning and 
personalized learning are not synonymous, they typically are conceptualized as comorbid types 
of learning. This report describes the terms below, and for the purposes of this document, it 
occasionally uses them interchangeably. 


Key Questions About Blended 
Learning 


What is Blended Learning? 


Blended learning is commonly conceptualized as the combination of both face-to-face and online 
or digital learning environments intended to afford teachers and students flexible opportunities in 
instruction, application, learning, and assessment. Blended learning environments, also called 
hybrids, provide some degree of student control over pacing, learning trajectories, timing, and 
place of learning, and provide the opportunity for the inclusion of real-time data. The Clayton 
Christenson Institute for Disruptive Innovation’ articulates four necessary components of 
blended learning (Staker & Horn, 2012): 
¢ Teaching and learning occur within a formal education program (e.g., K-12 for a high 
school diploma or certificate); 
¢ Online or digital delivery of content and instruction is a component of the program; 
¢ Students have some level of control over instructional time, place, path, or pacing; and 
¢ Part or all of the instruction is delivered away from home in a supervised, brick-and- 
mortar location. 


Based on its research on the implementation of blended learning programs, the Christensen 
Institute developed a typology of blended learning models with differentiation based on when 
and under what circumstances instruction and learning occurs in online versus face-to-face 
environments. These models are: (a) rotation, in which students rotate, typically on a rather fixed 
schedule, between digital learning, small-group, full-group and other modalities; (b) flex, in 
which the majority of instruction occurs in a digital environment—with some offline activities 
and face-to-face support provided on a flexible, adaptive, as-needed basis; (c) a /a carte, in 
which a student takes a course completely online with an online instructor, though the student 
may use computers at a brick-and-mortar school site where they take other traditional or blended 
courses; and (d) enriched virtual, in which a student takes a course primarily online, but must 
attend occasional face-to-face instruction with the teacher (Horn & Staker, 2014). 


However, as the International Association for K-12 Online Learning ((NACOL)’ explains, 
blended learning “is not teachers simply putting lesson plans online or content resources online” 
(Patrick, Kennedy, & Powell, 2013, p. 14). Instead, in blended learning environments, 
...1t is the magic of optimizing the face-to-face classroom with instructional models for 
personalized learning for teachers using online learning modalities and advanced 
technologies to accelerate and improve individualized learning experiences for each and 
every student, with real-time data on exactly how well each student is progressing. (p. 15) 


* The Clayton Christensen Institute is a nonprofit, nonpartisan think tank that promotes improved education through 
“disruptive innovation” that makes products and services simple, convenient, accessible, and affordable (see 
www.christenseninstitute.org). 

> iNACOL is an international nonprofit organization with the mission to support and transform K-12 education 
policy and practice to advance personalized, learner-centered, competency-based, blended and online learning (see 


www. inacol.org). 


Therefore, blended learning is typically conceptualized as a form of instruction that provides—to 
varying degrees—individualized, differentiated, assisted, and personalized learning 
opportunities. Individualized refers to adjusting the pace of instruction according to students’ 
individual readiness to learn; differentiated refers to the use of different instructional approaches 
according to student need while keeping learning goals the same; assisted refers to adjusting or 
accommodating instruction according to student need, ability, or disability; and personalized 
refers to all of the above, also including taking into account the learner’s unique interests and 
abilities (Grant & Basye, 2014). 


Affordances, Constraints, and Trends Pertaining to Blended 
Learning Environments 


Affordances 

Achieving the aims of personalized learning through blended learning is a formidable goal. To 
do so, iNACOL (Patrick et al., 2013) posits several affordances that should be incorporated into 
blended learning models’ combined digital and face-to-face learning environments. 


Highly personalized, customizable, and student-centered. The learning environment provides 
students some control over how they learn. Instruction is differentiated, pacing is flexible, and 
interventions and supports for students are available on demand and when needed. Learning 
environments should respond to each student’s needs and interests, be developmentally 
appropriate, and be informed by data and based on the science of teaching and learning. 


Competency-based and rigorous. Learning objectives are explicit, measurable, and align with 
standards of college and career readiness. Students should be able to demonstrate their mastery at 
any time, without having to adhere to a rigid instructional pacing schedule. 


Data-driven. Ynstruction to facilitate student learning should be informed by frequent and 
varying forms of meaningful formative and summative assessments (e.g., embedded, 
performance-based, project-based, and portfolio). These data can be delivered in real-time. 
Additionally, other sources of diverse data are made possible. An essential component here is 
that the technologies provide both teachers and student access to a wide variety of data that are 
made possible by those applications. 


Equitable and accessible. Students are afforded access to courses, experience, and teachers that 
they otherwise would not have (e.g., through remote communication such as video conferencing) 
as well as multiple methods of instruction and digital content. For example, online learning may 
benefit youth in areas with fewer opportunities to access high quality instructional supports, such 
as rural students or those with medically fragile conditions who learn from home (Aspen Institute 
Task Force on Learning and the Internet, 2014; Fernandez, Ferdig, Thompson, Schottke, & 
Black, 2016). At best, instructional methods are developed using research-based methods to meet 
the needs of learners with a range of abilities and disabilities, giving all students access to 
learning through multiple means of representation, expression, and engagement (Hashey & Stahl, 
2014; Meyer, Rose & Gordon, 2013). 


Staffed “flexibly” but capably. The way that teachers, outside experts, community resources, 
home mentors, online tutoring, and technical support are staffed permits instruction and learning 
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to happen any time. Doing so requires opportunities for educators to build a common 
understanding of proficiency and effective communication about past and present learning data. 
The possibilities afforded by blended learning are stimulating growing interest in effectively 
using technology to support appropriate teaching, learning, and assessment environments. 
Toward this end, student information and competency-based assessment and learning 
management systems can provide relevant, easily accessible, and shared data to support students, 
teachers, and schools. With sophisticated tools, educators can harness information from a wide 
variety of sources, including from the “digital ocean” (DiCerbo & Behrens, 2014), to achieve the 
goals of personalized learning (Bienkowski, 2014; Bienkowski, Feng, & Means, 2012). For 
example, innovations in flexible learning technology aspire to incorporate information about 
individual students such as prior academics, prior and real-time cognition, on-task and 
perseverant behaviors, among other data to adjust digital learning environments for individual 
students (Abell, 2006; Fletcher, Scaffhauser, & Levin, 2012). Developing software that can do 
these things automatically can help offload a great deal of the work that typically falls on 
instructors’ shoulders. Progress is being made in research on data analytics and instructional and 
assessment software that can provide an understanding of how a wealth of formative, real-time 
data about students’ progress can inform instruction. 


There are several key advantages here. First, teachers and students save time through the 
provision for access to data and resources that may or may not be available without the 
technology in a cost-effective manner. Second, the diversity of data provides teachers and 
students with data that might not be available in traditional classrooms. Third, the data arrive in 
real-time, providing a tighter feedback loop among instruction, assessment (or the like), data 
collection, data analyses, and result interpretation. This iterative cycle means that data collection 
and feedback are much more tightly woven to the teaching and learning process. 


Constraints 


The degree to which the affordances of blended learning can be attained likely varies 
considerably from school to school and even classroom to classroom. This may be due in part to 
variations in design, fidelity of implementation, and other environmental constraints. For 
example, educational leaders may have different goals with regard to the degree of 
personalization desired or the ways that technology and data will be used—and the learning 
environments they implement reflect these specific visions. Variations in blended learning’s 
affordances also may be due to limitations or constraints imposed by a school system’s existing 
infrastructure. Policy constraints; technology challenges; limitations of digital learning programs; 
and emerging teacher and student competencies are some of the issues that can impact 
implementation of blended learning. These constraints are described briefly below. 


Policy challenges. Research has begun to attend to policy environments that may constrain or 
facilitate adoption and implementation of blended, online/digital, and personalized learning. 
These include, for example, changes to the requirements and structures of course in-seat time and 
state and district assessment schedules (Redding, 2016). In blended learning environments with a 
highly personalized learning structure, a student should be able to advance based on 
demonstrated mastery or competency. However, most school systems have not relinquished the 
concept of Carnegie units needed to earn a course credit toward high school graduation, defined 
by seat time or hours in class. The conventional model of advancement relies, in part, on pacing 
guides and fixed assessment schedules for courses. This means a student surging ahead may not 
have access to or may be discouraged from pursuing more advanced learning, especially if that 


learning is designated for a subsequent course. Conversely, a student who is struggling with key 
concepts may nonetheless be advanced to maintain the proscribed pace of the curriculum, 
exacerbating performance gaps over time. Maine, Oregon, New Hampshire, Iowa, and Ohio are 
among the pioneering states establishing competency based education systems relinquished from 
these time-bound constraints with some success (Freeland, 2014; Patrick & Sturgis, 2013; Wolk, 
2015). 


Technology challenges. Technology issues that impede data use in blended learning 
environments include challenges such as limited internet connectivity and broadband; difficulty 
obtaining sufficient computer hardware; and insufficient data and analytic systems for collecting, 
synthesizing, analyzing, and reporting student learning and other data. Not all communities and 
students have equal access to these technology infrastructures outside of or even in school, and 
the disadvantage of the “digital divide” is most apparent among those who live in or attend 
schools in economically distressed areas (Dolan, 2016). At an entry level, schools use technology 
in limited ways, such as for electronic gradebooks and tracking student demographic data, 
attendance, and credits (CCSSO, n.d.). Sophisticated content management systems, learning 
management systems, portfolio systems, data dashboards, and so on may not exist, and if they 
do, they are not likely to be fully integrated, accessible, or aligned. They may not be 
interoperable; that is, have the capacity to share data. At the most advanced levels, student- 
centered systems should be in place to assess learning, integrate data, produce data analyses, 
showcase authentic work, and manage learning and assessment resources adapted to personalized 
content. Establishing these basic infrastructures is no easy task, and most technology tools are 
used for more rudimentary management and resource sharing (CCSSO, n.d.). Some research 
indicates K-12 online and blended programs are not prepared for the collection and analysis of 
data required to inform instruction, and data from various digital learning management and data 
systems have proven difficult to manage and interpret (Ferdig & Cavanaugh, 2011). 


Limitations of digital learning programs. Digital learning environments that lack coherence and 
fail to align with the school system’s adopted curriculum standards can pose challenges to 
blended learning. “Off the shelf” digital instructional technology developed by third-party 
vendors, for example, may be designed to automatically adjust the content, order, and pacing of 
the learning environment for individual students according to built-in formative data analytics, 
but these adjustments may not necessarily match the school system’s content standards, order, 
pacing, or desired rigor. The value of digital learning technologies relies, also, on their research- 
based design; however, there is some evidence that currently available digital curricula may not 
be better than what is found in printed textbooks (Choppin & Borys, 2016), and they may not be 
appropriately designed for or accessible to students with certain disabilities (Basham, Stahl, 
Ortiz, Rice & Smith, 2015; Smith, 2016). Other types of digital learning software may allow (or 
require) teachers or students themselves to adjust these features of the digital environment, in 
which case efficient and valid formative assessment systems aligned with the adopted standards 
are essential for informing these decisions. In instances when teachers are not provided with a 
fairly comprehensive set of digital curriculum materials at the outset and instead are asked to 
curate digital materials for personalized learning, “foraging” behaviors have been observed that 
may result in an instructional curriculum with diminished coherence (Selwyn, 2007). 


Instructor and student competencies. Blended learning can be complex environments for 
educators to implement. They require a great deal of expertise in the content area, pedagogy, and 
management of the digital and face-to-face environments, as well as sophisticated use of data to 
drive students’ learning (Kennedy & Archambault, 2013). Professional development may be 


required at the onset and in an ongoing way to address teachers’ skills in these areas. With regard 
to students, blended environments may be an easy fit for learners who have experience and 
demonstrated competency with independent work. However, it is unlikely that all students enter 
blended learning with the requisite prior knowledge of the content area, skills for staying 
engaged, and metacognitive and self-monitoring skills to be able to succeed. Rather than 
assuming students enter the blended learning environment equally ready for success, some—or 
perhaps many—-students may benefit from support with developing and applying these skills 
(Redding, 2016). 


Trends 
Although the history around blended learning is not particularly long, it is possible to discern 
patterns and trends that have been emerging as implementation and research are evolving. 
DreamBox (2014) identified 10 trends: 

¢ The deeply student-centered learning experience; 

¢ Soaring numbers of digital learners; 

¢ Supporting standards and higher-order thinking skills; 

¢ Realizing benefits for both students and teachers; 

¢ Data-driven instruction to personalize learning; 

¢ Personalized learning accompanied by a lean, blended, iterative approach; 

¢ Productive gamification; 

¢ The mobile world is where learners live now; 

¢ Bring Your Own Device (BYOD) is here and key to active three-screen days; and 

¢ More broadband please. (p. 3) 


Some of these trends are more relevant to the work described here. For example, creating a 
student-centered environment requires more active engagement on the part of the learners. 
Similarly, a focus on higher-order skills requires students to engage in deeper cognitive activities 
with more creativity and less passive and rote learning. The engagement impacts not just 
students, but also affects the changing role of the teacher. Blended environments tend to be more 
active, both in the classrooms and with the opportunities for virtual connections. Blended 
environments also provide data that heretofore might not be readily collectable or available in 
real-time that allow for customizable teaching and learning experiences. Finally, with the 
proliferation of mobile devices, emerging technologies, and more ubiquitous connectivity, 
blended environments provide for more flexibility throughout the teaching and learning process. 


Blended Learning in the National Context 


When the U. S. Department of Education (2016) released its most recent national education 
technology plan, blended learning and related components loomed large. The plan was divided 
into five sections — Learning, Teaching, Leadership, Assessment, and Infrastructure — that play 
an important role in the implementation of blended learning. The plan emphasizes the role of 
technology in the design of teaching and learning activities and notes that: 


Teachers collaborate to make instructional decisions based on a diverse data set, 
including student and teacher observations and reflections, student work, formative and 
summative assessment results, and data from analytics embedded within learning 
activities and software aided by real-time availability of data and visualizations, such as 
information dashboards. (p. 41) 
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The plan also emphasizes the role of technology for assessment, providing new and more 
innovative means by which to measure student learning. The plan lays out that the next 
generation of assessments will be: embedded in learning, universally designed, adaptive to 
learners’ ability and knowledge, with feedback given in real time, and with enhanced item types 
that can measure complex competencies. Embedded assessments that are technology-infused can 
provide real-time data and immediate feedback. Whether through assessment platforms or data 
dashboards, such data have the potential to provide more meaningful and timely information to 
students and teachers about learning progressions. And the data will be more diverse, in-depth, 
and informative for the teaching and learning process. In combination, the diverse data and the 
immediacy of feedback, will enable more highly adaptive learning experiences made possible by 
the technology. 


Recognizing that the current state of data systems makes interoperability a challenging prospect 
at best, the plan stresses the need to overcome this issue so that data from multiple sources, 
including formative and summative assessments, can reside in one repository. The single silo 
will facilitate easier access for teachers, students, data teams, and administrators. It will enable 
the triangulation among diverse sources of data to obtain a more comprehensive picture of the 
students. 


Having the technology infrastructure in place is important, but so too is the provision for the 
human infrastructure; that is, that educators must know how to use the data effectively and 
responsibly (Data Quality Campaign, 2014; Mandinach & Gummer, 2016). They must be data 
literate. Having the students also involved in the examination of data also is a recommendation 
posed by the Practice Guide on data-driven decision making released by the Institute of 
Education Sciences (Hamilton, Halverson, Jackson, Mandinach, Supovitz, & Wayman, 2009). 


Implementation Research 


Because the field of blended learning in K-12 environments is relatively new, its research and 
evidence base is also emerging. In almost all the existing implementation studies, the districts 
and schools under investigation were themselves new to blended learning, innovating, and 
evolving at a rapid pace. Therefore, because of blended learning’s relative newness in K-12 
settings, the field does not yet have information about what longstanding, established, stable and 
sustainable models might look like. Available studies focus on a variety of implementation 
issues, ranging, for example, from district level leadership and organizational, administrative 
structures that support it; to financial models; to instructional issues; classroom management and 
even physical arrangement of students. Three studies of blended learning implementation are 
highlighted here. They attend to how instructional decisions are made in blended learning 
environments, in particular to how data are used. These studies were chosen because they inform 
the field about key components for effective or less effective implementation and as related 
directly to the research findings reported here. 


RAND Studies 


RAND conducted a study of 32 Next Generation Learning Challenges (NGLC)* schools that 
were innovating and implementing personalized learning environments—most with blended 


*NGLC was founded in 2010 by EDUCAUSE in partnership with the Council of Chief State School Officer 
(CCSSO), the International Association for K-12 Online Learning (NACOL), the League for Innovation in the 
Community College, and the Bill & Melinda Gates Foundation (see http://nextgenlearning.org). EDUCAUSE is a 
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learning components (Pane et al., 2015). The implementation study included schools serving 
mostly urban, minority students from low-income families in a variety of grade level 
configurations (Pre-K-5, K-8, 6-8, and 9-12). 


The study found more than three quarters of the schools were implementing some degree of 
flexible or “multiple paths” to progress through course content, typically as students rotated 
through small group, large group, and independent work within a classroom. Much of the 
instructional time was “technology-led” or “technology-facilitated.” Ways technology was used, 
however, varied widely, with half or more teachers reporting its use for routine tasks such as 
reading, watching videos, using online reference materials, and using structured curriculum 
materials. Fewer, about a third or less, reported using technology for more complex tasks such as 
solving problems, collaborating with other students, or using adaptive learning software for 
problem-solving and adjusting parameters of simulations. 


Several challenges emerged with the implementation of personalized learning. Flexibility with 
students’ pacing and advancement according to mastery was reportedly hindered by constraints 
such as the perceived need to emphasize grade-level content and the desire to ensure students 
were making progress toward grade level standards through traditional accountability systems 
and metrics such as course completion and standardized testing. Schools that did not organize 
students by traditional grade levels and that were not constrained by these accountability systems 
were more likely to implement flexible pacing and advancement. 


Overall, teachers in the NGLC schools were implementing various features of personalized, 
blended learning, and used data to do so. Compared with teachers from more traditional schools 
serving similar students, teachers in NGLC schools were more likely to report their data and data 
systems were of high quality and useful. They also were more likely to report they used a variety 
of data and data sources to understand student progress and inform instructional decisions. 
NGLC teachers reported that they had access to relevant information from tests, quizzes, and 
projects as well as non-achievement data. Over half of the teachers reported that they used data 
such as these at least weekly to identify each of the following: which students had attained 
mastery, which students mastered specific content or skills, and which students needed additional 
help. However, although teachers said they had access to a great deal of student data, 61 percent 
said they needed help applying this information to instruction. 


SRI Studies 


SRI and FSG conducted an extensive series of implementation studies (Bernatek, Cohen, Hanlon 
& Wilka, 2012c; Murphy, Snow, Mislevy, Gallagher, Krumm, & Wei, 2014). This study series 
examined implementation and outcomes of five charter management organizations (CMOs) 
overseeing schools with a range of grade level configurations (K-1, K-5, K-8, and grades 9 and 
10). In these CMOs, typically, students rotated from digital learning at in-class or computer-lab 
stations to face-to-face small instructional groups led by a classroom instructor and other 
independent learning stations. In this way, students in the digital environment were not far from 
live, personal support when needed. 


Instructors across sites articulated needs for digital learning programs offering a combination of 
features, including: 


nonprofit association whose mission is to advance higher education through innovative use of information 
technology (see http://www.educause.edu). 


* Comprehensive and aligned with Common Core standards; 

¢ Adaptive, meaning they were programmed to adjust automatically to students’ progress 
and needs; 

¢ Were “assignable,” meaning the teacher could make adjustments to the content, level, or 
sequence of the program for each student; 

¢ Provided valid reports of student engagement and learning, including daily formative, 
interim, and summative reports mapped to standards; and 

¢ Allowed student data to be integrated into the school’s data system for easy access and 
use, particularly when multiple programs were used. 


The study noted that educators typically underutilized student performance data from online 
programs to inform instructional decisions, in part because the instructional software programs, 
provided by outside vendors, did not offer all of these features. For example, software programs’ 
data dashboards were not integrated into the school’s larger data system, a challenge further 
exacerbated when performance measures, progress measures, and dashboard data were not 
aligned with the content standards that the schools used. In some cases, educators questioned the 
validity of what should have been very useful formative data provided by the digital instructional 
programs, so teachers turned to their own formative assessments to measure student progress and 
mastery. The vast majority of teachers surveyed across all five CMO sites reported it was “very 
important” to receive proper training on accessing and interpreting student progress reports 
provided by the online instructional programs and using it to inform their instruction. 


Data use may be enhanced by blended learning environments, a focus of particular importance to 
work reported here, Yet Murphy and colleagues (2014) noted a number of factors that impeded 
the use of data in the blended learning environments they examined. The study found that the 
technology-generated data were not well used. This finding might be due to the quality of the 
data dashboards, poor accessibility, the difficulty in interpreting the data, a lack of alignment of 
the data to the standards and curricula, and a lack of trust in the data. A major impediment 
seemed to be a lack of interoperability among diverse sources of data, making it difficult for 
teachers to triangulate among these silos of data. Time was also an impediment. Teachers simply 
did not have the time to examine the data in depth. 


Despite the impediments around the data, the study reported benefits to the teaching and learning 
process. Teachers were able to customize learning opportunities for the students, particularly 
those most in need. The teachers used adaptive online materials, self-paced programs, and small 
group instruction. 


Bingham Study 

Most recently, Bingham (2016) conducted a qualitative study of a blended learning charter high 
school in a large urban area that served primarily low income and minority students. Data 
collection focused on the inaugural cohort of ninth graders and their instructors and school 
leaders. The intention was for students to progress through courses at their own pace, engaging 
primarily with the digital curriculum, and asking for help with on-site teachers when needed. 
Coursework was to be accelerated when the student demonstrated mastery, or slowed down to 
fill in necessary gaps of knowledge and skill. The vision was that teachers would serve as 
facilitators and coaches, providing targeted support to individuals or groups as needed. To free 
up teachers for their new role as facilitators and coaches in the classroom, the selected digital 


curriculum was to readily provide a range of learning activities, and was to perform most of the 
work of assessing student progress and mastery. 


In contrast to this vision, Bingham observed the digital curriculum did not meet instructional and 
assessment expectations for a variety of reasons, such as leaders’ and teachers’ insufficient 
instructional planning, organization, and management of blended learning classrooms. The 
digital curriculum did not offer and adjust to the full range of students’ needs, particularly those 
who did not have necessary skills or knowledge such as vocabulary and reading to work 
independently with the digital curriculum. In some cases, the curriculum was not rigorous 
enough. Consequently, “teachers could neither rely on the digital curriculum to provide students 
with opportunities to learn the material nor expect the information provided by the digital 
curriculum to be accurate” (p. 20). To cope with the demands of individualizing and 
personalizing students’ learning environments, teachers felt they were “drowning digitally,” and 
almost all returned to low-tech strategies for instruction and assessment. By the end of the year, 
classrooms looked more like traditional classes than the intended innovative, blended and 
individualized learning environment that was intended. The study concluded that “if an online 
curriculum is expected to bear the responsibility of assessment and data production, the online 
curriculum must first be vetted to ensure the assessments are rigorous and the data is accurate” 


(p. 27). 


Outcomes Research 


Research on the outcomes of K-12 blended learning environments, particularly rigorous, 
independent, peer-reviewed research, is scarce (Cole, Kemple & Segeritz., 2012; Means, 
Toyama, Murphy, & Bakia, 2013). The results are quite mixed. A small number of studies find 
positive impacts of specific formats of online learning, especially when it promotes collaboration 
and self-reflection among students (Bakia, Shear, Toyama & Lasseter, 2012). Female students 
were found to have improved math knowledge and problem-solving ability after engaging in 
digital mathematics instruction (Arroyo, Burleson, Tai, Muldner, & Woolf, 2013). Other 
researchers found a blend of online and school-based instruction is more successful in improving 
student outcomes than an online-only model (Bernard, Borokhovski, Schmid, Tamim, & Abrami, 
2014; Means et al., 2013) and work “marginally better” than traditional classroom instructional 
models (Bernard et al., 2014). Other research finds blended learning has neutral or negative 
effect on student achievement (Cole et al., 2012; Margolin, Kieldon, Williams, & Schmidt, 
2011). Typically, authors cautioned that the reasons for the findings were unclear. 


At this time, few research results from outcomes evaluation and research are available to inform 
best practices. Some of the challenges with conducting rigorous outcomes research in this area 
include: 
¢ The great variation in instructional models, use of digital learning software, nature of 
face-to-face learning, degree of personalization, content areas, and special populations 
(e.g., students at risk of failure; returning drop outs; students with disabilities; and 
advanced, accelerated learners (Bernard, et al, 2014; Lowes, 2014); 
¢ Identifying the appropriate comparison group, because almost all traditional classrooms 
now incorporate some type and degree of digital learning, and almost all online-courses 
also incorporate interaction and assistance from an instructor even if that instructor is 
distal rather than face-to-face (Lowes, 2014); and 


¢ Challenges in conducting experimental design studies, given the organizational 
constraints of states, districts, and schools implementing blended learning and other 
learning environments (Lowes, 2014). 


The outcomes analysis of the RAND study is one of the first of its kind to offer a comparison 
group analysis of student achievement outcomes in blended and personalize learning programs 
implemented at scale (Pane et al., 2015). Analyses found that, overall, students in 62 
“personalized learning schools” performed .27 standard deviations higher in mathematics and 
0.19 standard deviations higher in reading on Measures of Academic Progress (MAP) 
assessments compared with students in a matched sample. However, effect sizes varied 
considerably across schools. The nature of “individualized learning” also varied considerably 
among schools, and analyses of certain instructional components suggested that two out of three 
components were present in schools with the highest effect sizes: 

¢ students were placed into flexible groups based on data; 

¢ student learning spaces were supportive of personalized learning; and/or 

* students were included in data based decision-making for learning. 


Due to limitations in the design of the study, authors have been cautious about generalizing these 
findings to other blended learning environments. 


In sum, research on K-12 blended learning and personalized learning environments may offer its 
greatest insights when it answers “Under what conditions does blended/personalized learning 
work?” (Ferdig & Kennedy, 2014; Pytash & O’Byrne, 2014). 


Methods 


This study used a purposive sample of three charter schools from one CMO in Arizona. The 
Pima Prevention Partnership is comprised of one high school in Phoenix, the Arizona Collegiate 
High School, and a high school and middle school in Tucson, the Pima Partnership High School 
and the Pima Partnership Academy. These three schools serve highly challenged youth and their 
families. 


The Pima Partnership Academy serves over 100 students in grades 6 through 8. The Pima 
Partnership High School serves up to 280 students in the Tucson area. The Arizona Collegiate 
High School serves roughly 200 students in the Phoenix area and is designed as a college 
preparatory institution. 


These schools were targeted because of their history of infusing diverse technological 
applications into their educational solutions to reach the challenged students. The schools also 
have strong building leadership focused on the use of technology and data to support teaching 
and learning, reflecting the recommendations from the Institute of Education Sciences practice 
guide on data-driven decision making (Hamilton et al., 2009). 


The study focused on a core sample of teachers in each of the schools where the study tried to 
maintain continuity over two years to mitigate the teachers’ level of mobility. The study also 
included building administrators who provide strong leadership in terms of technology infusion, 
data use, professional development, and a commitment to enhance teaching and learning. 
The study used multiple measures to triangulate data sources. The measures included interviews 
with teachers and administrators, a survey, multiple classroom observations, an artifact protocol, 
and a data literacy protocol. Each of these measures focused on the collection of data to address 
the following: 
¢ What are the technological applications being used in the classroom, the school, and 
through mobile technologies to enhance teaching and learning? 
¢ What are the affordances of these technologies that provide data to teachers and students 
to enhance teaching and learning? Are these data unique to the technological 
applications or are they readily available without the technology? 
¢ What other resources or artifacts do teachers use to enhance their instruction and 
classroom practices? 
¢ What are the supportive resources provided by the school administration to make possible 
the blended learning environments? 


Data were collected over a two-year period with several visits to the schools. During that time, 
the schools had experienced some turnover in the teaching staffs so additional teachers were 
added to the sample. 


Results 


Results from the multiple sources of data collection — interviews, artifact protocol, classroom 
observations, data literacy protocol, and survey - are reported based on the specific method. A 
topical discussion will follow in the concluding section. 


Interviews 


Interviews were conducted with all participating teachers and administrators from the three 
schools during the first site visit. As new teachers joined the sample, interviews were conducted 
with them. A second set of interviews was conducted with the teachers on the final site visit to 
discuss progress and changes over the course of the two academic years. The interviews were 
conducted in a semi-structured format, meaning that although there were specific questions to 
address, the protocol allowed some flexibility for modifications based on prior responses. The 
overarching topics included: blended and personalized learning environments, technology, data 
sources, teachers’ backgrounds, data use and beliefs, and information about the school and its 
support structures. The interview protocol can be found in Appendix A. 


Administrator Interviews and School Overviews 

The students served by the Pima Prevention Partnership are primarily Hispanic and African- 
American. Most students come from impoverished homes. Some students are in a group or 
custodial home. Over 90 percent are on free or reduced lunch programs. Many students have 
attendance issues due to long commutes. The students may also lack supportive home 
environments, with parents tending to not be engaged in their children’s education. Because of 
the challenged population, the explicit vision for all educators is respect for student needs. Socio- 
emotional and coping skills loom large. Educators hold high expectations for all students and 
strive to have them attain college. They do this through a nurturing and flexible environment that 
attempts to customize to the evolving needs of the students. Technology plays a major role in the 
customization. 


According to the three principals, Pima Prevention Partnership has made technology integration 
a top priority for all of the schools. The use of technology is seen as a way to bridge the gap to 
deal with relevance because the students have all grown up with technology. The educators are 
using technology to provide resources to meet the needs of the students. For example, there are 
some classroom sets of books, but much more use of online books. Spanish books are available 
online. These books use the affordances of the technology through video and the audio and are 
viewed as more engaging media. In some sense and to some of the students, the technology may 
be considered Edutainment, because the students have to be entertained to be engaged. 
Administrators do not believe that this takes away from the education value. They use 
technology to supplement and enhance, not supplant instruction. 


Because the students served by the three schools come from challenged environments, the 
educators must find creative ways to engage and keep the students from dropping out. For one of 
the high schools, the graduation rate has progressed from 24 percent to 36 percent, and then to 50 
percent. Students were at 6" grade math and 4" grade reading levels, but the administration 
recognized that it will take time and effort to fill in the gap. Half the high school’s population is 
over 18, with a very high mobility rate and a low average daily membership. Yet the 


administration plans to remain resolute about maintaining high but realistic expectations. 
According to the principal, “If you drop the bar, you are not helping the kid.” 


They see leveraged technology as a way to bridge the gap. They use various technologies to 
enable students to be connected from any where at any time. The technologies are diverse. All 
classrooms have Smartboards, projectors, COWs (computers on wheels — 
30 notebooks), laptops, responders/clickers, graphing calculators, and 
smart tablets. They use some of the state technology, including ADE 
Connect and AZDash to provide a different level of data. They use the drop the bar, you 
Galileo assessment system from Assessment Technology Incorporated : 

‘ . are not helping the 
(ATI to provide formative assessments and to have teachers and students kid.” 
examine test results. This helps to engage students in data conversations. 
Students are expected to accept responsibility for their own learning. 
Students are also expected to plot their own data and monitor their progress, understanding what 
they need to do to improve. The philosophy is that data drive the discussions. 


Accodring to one 
principal, “If you 


The administrations provide many resources for teacher professional development around 
instruction, curricula, data, and technology. They bring in professional development providers on 
a regular basis, send teachers to conferences as needed, and provide professional development on 
Wednesday afternoons, where technology and data use are expected. 


The schools use non-test data to evaluate teachers. Because the attendance rate sometimes falls 
below 50 percent, the administration believes that you cannot define a teacher when kids are not 
in class. Students come from a 50-mile radius to the campuses, causing a host of realistic 
problems. Transportation is a financial and logistic problem so they provide students with public 
transportation passes. The Partnership realizes that the families care about the kids but 
circumstances are challenging so the Partnership tries to help the families in any possible way 
and to create opportunities for the students to stay in school. Although the majority of the 
students have Internet access through a cell phone, programs are designed so that they can stay 
on campus to get their work done. Most students do not have a home environment that promotes 
learning so homework is about learning and mastering the standards. Faculty members stay late 
to work with the students. There is a conscious effort for teachers to model good behavior and 
ethics. The result is that the schools have moved from a state grade of D, to a C in Year 2, and to 
a B in Year 3. 


The schools try and recruit data and technologically savvy teachers and then provide resources to 
create a community around data and technology use through data teaming and professional 
learning communities. Two hours each week are devoted to teaming and to professional 
development. 


The schools now have a robust data collection system and process. There is a center server and 
data system, yet there are still data silos given the diversity of technological applications. The 
principals are all data savvy and are constantly working to improve the technological and human 
capacity of their staffs. There is one central “data guy” that serves all three schools. He is the go- 
to person for all things data-related. He handles data from Galileo as well as the changing 
landscape of testing in Arizona (from AIMS, to AZMerits, and now to local control of testing). 
Data are collected through Galileo, the state system, SchoolMaster, Gradebook, AZDash, and 
other applications. Diverse data are collected, not just student performance: family contact 
information, medical (vaccination, special needs), attendance, tardiness, early dismissal, 


transcripts, missing assignments, grades, languages, family circumstances, growth reports, and 
other data. There is a FamilyLink for families and students to gain access to the system. 


Most staff have more than a passing knowledge of data and technology, due in a large part 
because of extensive professional development opportunities. Yet, because of teacher mobility 
due to the intense and demanding environment, there is a constant need for more professional 
development, particularly for newer staff. Administrators are seen as highly supportive of teacher 
needs in terms of training, professional development, and the provision for resources. 


Teacher Interviews 

The teacher interviews yielded information about four general topical areas: teacher background 
and their beliefs around technology and data use; components of blended learning environments; 
technology; and data use. 


Teacher Background 

The sample of teachers ranged from fairly new teachers (second year teaching) in the first year of 
the project to experienced ones (22 years). The average number of years taught was 9.5. One 
teacher had come from industry, another was alternatively certified through Teach For America, 
and another was trained internationally (the UK). 


All but two of the teachers reported having received training in the technologies used in their 
schools and around data use. The technology vendors provided targeted training on the use of the 
Smartboards and the Galileo assessment system. The Arizona Department of Education provided 
training on their data system applications. All but one of the teachers reported that professional 
development, conference attendance, and other learning opportunities abounded for them. 
WestEd provided professional development on a variety of topics. 


In terms of their beliefs, all of the teachers reported that they believed that using data has the 
potential to improve instructional practice and help to meet their students’ needs. They all 
recognized the importance of data and provided rationales for how data can be used effectively 
in their practice. A number of teachers mentioned that the use of data helps them to consider 
what to teach, what to reteach, and allows them to plan their 

instruction. Several teachers commented that data drives decisions and One feacher 

their practice must be grounded in data. One teacher commented that commented, “Data 
data must be more than standardized tests because they are consumed must be more than 
by these results. Another teacher noted the importance of needed standardized tests... 
diverse data and multiple data points. Some teachers focused on the we are consumed 
challenged group of students they teach and how data use allows them by these results.” 

to focus on the whole child, fills in the gaps from solely looking at test 

results, and provides a sense of responsibility and pride in meeting the diverse needs of their 
students. Data tell the teachers where to go and what gaps need to be addressed. Two teachers 
also mentioned the need to involve the students in their own data use. The take-away message 
was that the teachers could not meet the needs of the students without data and the technologies 
made possible teacher and student data use. 


When asked about data literacy, most teachers considered themselves to be data literate. Two 
teachers thought they were not data literate and one was unsure. Many reported that they were 
but were constantly improving to be more data literate. When asked if they have ever taken a 


course on data use, four said yes. One was the internationally trained teacher; one noted a 
statistics course; one took a graduate level course; the other noted the assessment course. When 
asked to identify data literacy skills, the teachers’ responses were quite varied and telling. Some 
were articulate whereas others seemed unclear. Many focused on the interpretation of data in 
terms of their instructional practice. Table 1 provides some of the responses. 


Table 1. 

Teachers’ Responses: Identifying Data Literacy Skills 

Adjusting and monitoring instruction Modifying instructional plans 
Determining if students meet the Statistical awareness 

standards 

Diagnosing Synthesizing different data 

How to collect data Understanding bad data 

How to filter what data mean Understanding patterns 

How to know where the kids are Understanding what data mean 

How to mine data Using email 

How to query data Using formative and summative assessments 
Hot to read data Using historical data 

Knowing what | do with the data Using multiple assessments 

Looking at more than snapshot data Using qualitative and quantitative data 


It is clear from these responses that some teachers have at least a basic understanding of some of 
the skills of data literacy (Gummer & Mandinach, 2015; Mandinach & Gummer, 2016), whereas 
others struggled with the concept. However, the list is only a subset of the skills and knowledge 
needed to use data effectively. 


Blended Learning Environments 

Two key components to the effectiveness of the blended learning environments observed in the 
three Pima Prevention Partnership schools were the explicit visions for the use of technology and 
data, and the strong leadership to support the environments. Every teacher noted that there was 
an explicit vision for the use of technology. The vision included innovation, meeting the needs of 
diverse learning through many media, getting technology in the hands of teachers and students to 
enhance the teaching and learning process, providing a supportive environment where everything 
is data-driven, and making the schools paperless. The foundational mission is to help all students 
to live to capacity and succeed. School leaders embrace the vision and model it to all staff and 
students. 


Teachers were asked how their blended classrooms differ from more traditional ones and what 
the value-added is from these environments. All of the teachers mentioned that blended learning 
creates an environment that is more active and engaging for students and requires more hands-on 
activities. The environment seems to be less formal because it lends itself to more project-based 
and learner-centered activities. Teachers’ notes that the blended environment gives these 
challenged students in particular access to a digital world to which they might not otherwise have 
access. It makes them digital citizens. It therefore has the capacity to level the playing field and 
provide a door to learning for even the most challenged students. 


For teachers, the blended classroom provides significant time saving through the learning 
technologies, particularly the Galileo assessment system. Galileo provides immediate feedback 
to teachers and students that would not be possible without the technology, particularly in terms 
of the drill-down capacity for item analyses. It allows both teachers and students to know where 
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students are in relation to state standards and helps the teachers to formulate and plan instruction. 
The data the system generates helps to drive instruction by providing information from which 
teachers can prioritize learning goals. The system produces understandable metrics and graphical 
displays of results and performance trends. It helps the students with self-monitoring skills as 
they map their learning progression and progress. The system can provide solid evidence of 
learning from reliable measures. 


For teachers, the blended environment provides a great deal of flexibility in how to provide 
instruction. There can be individualization with a total customization to each student’s needs. 
There can be small groups or full class instruction. The technologies allow teachers to know 
where each and all students are in terms of learning goals at all times. Student learning and 
growth can be monitored at global or fine-grained levels. This flexibility enables teachers to 
pinpoint students’ strengths, weaknesses, and learning needs. 


The blended environment stimulates better organizational skills and efficiency for both teachers 
and students through access to diverse data sources that inform the teaching and learning 
process. Regardless of discipline, the data provided by the technologies enables teachers and 
students to gauge learning progress through rapid or immediate feedback. Teachers report that 
classrooms are more flexible, rigorous, more visual, more research-oriented and project-based, 
and less paper-oriented. Teachers and students have any time and any where connectedness, 
allowing for extended hours for learning opportunities. 


Teachers report that students feel safe in the blended classrooms. The technologies somehow 
remove a level of fear from the students, especially when they are working in peer groups. 
Teachers also report that there is an increase in student success and evidence of enhanced 
learning. Students are more involved and engaged, have a clear understanding of expectations, 
therefore can take ownership for their own learning. 


Blended learning environments promote the use of diverse data because of the affordances of the 
technologies that might not be possible in more traditional educational settings. The affordances 
relate not only to the types of data but the feedback loop from which teachers and students are 
able to access the data. For example, The Galileo assessment system 

and the Smartboards provide quick turnaround data, data on which As one teacher noted, 
instructional modifications can be made almost in real-time. The “Real-time data are 
immediacy of the feedback and the depth and breadth of the Galileo powerful.” These data 
analytics through its drill-down capability are immense time savers help teachers to 

for teachers. They provide reports and results that would take understand students’ 
teachers hours to perform if they could perform such analyses at all. strengths, weaknesses, 
Teachers are also able to accumulate historical or longitudinal data to @nd misconceptions. 
trace trends, patterns, and progress over time. As one teacher noted, 

“real-time data are powerful.” These data help teachers to understand students’ strengths, 
weaknesses, and misconceptions in a manner that yields diagnoses that can be translated into 
instructional modifications. The performance-based data help to guide instruction and allow 
teachers to know what to reteach. 


Not to provide a skewed notion that all data are assessment or cognitive data, the teachers 
reported diverse data that are not necessarily yielded by Galileo. These include attendance, music 
performance and music theory, fitness and nutrition data, observational data, attitudes, absences, 
and demographics. There are other student performance data collected as well, those that are 
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discipline-specific and linked to classroom activities, content-related data, state assessments, and 
other types of measures. The foundation of all these sources of data is that they provide an 
understanding of the students, along with data such as contextual data. 


One of the components of blended learning environments is the changing role of the student. 
These environments place more responsibility on students for their own learning and monitoring 
their progress through self-examination of data. Essentially, students must become their own 
data-driven decision makers, one of the recommendations of the Institute of Education Sciences 
practice guide (Hamilton et al., 2009). Students must self-monitor. They look at their data and 
monitor their own performance. Because of this, students are more 


active, engaged, and motivated. They may even become Teachers report that 
competitive. They must take more responsibility and ownership students grow to 

for and pride in their own learning. Teachers report that many understand expectations 
students are receptive to the process, whereas some are less so. my monitoring their 


Teachers also report that students grow to understand expectations progress. They gain time 
by monitoring their progress. They gain time management skills management skills and 
and collaborative skills. One teacher uses a peer editing process collaborative skills. 

that requires collaboration and trust. Students see their success and 

build on that progress. The process provides increased transparency around the expectations and 
standards. Teachers also report that the students feel safe in this environment. They help to guide 
one another. The increased responsibility applies beyond the boundaries of the school because of 
the mobile technologies. Students communicate with teachers and other students through social 
media, email, texts, and the instructional portals any time and any where. 


One of the components of effective data use is for teachers to engage in collaborative inquiry 
through data teams and professional learning communities (Hamilton et al., 2009; Love, Stiles, 
Mundry, & DiRanna, 2008; Mandinach & Jackson, 2012). Each school had a math data team and 
a language arts data team. Teachers outside those disciplines also participated because of the way 
the curriculum was structured. For example, the science and history teachers used writing 
prompts that addressed certain standards. The music teacher in one of the high schools 
participated in both data teams because she taught the concept of fractions through musical notes 
and composition through required writing assignments. 


The data teams met at least once a week, if not more. Wednesday afternoons had protected time 
for teachers to meet and discuss student data. Teachers would analyze data and examine student 
performance. They would note students who were struggling and those doing well. They would 
discuss deficiencies by drilling down into data. They would use a data dialogue process with 
guiding questions to structure their analyses. The teams would analyze lessons and determine 
what needed to be retaught and how the instruction needed to be modified. The teaming process 
was based on an open and constructive exchange of ideas in a non-judgmental environment. 


Technologies 

Technologies are a major component in blended learning environments. The three Pima 
Prevention Partnership schools most definitely had diverse technologies infused in the 
classrooms and even made some available through remote and mobile devices. 

The schools were equipped with a multitude of technologies. Some technologies were classroom 
based, whereas others were mobile. There was an emphasis on mobile technologies to allow the 
students to have access any time and any where to instructional and assessment materials. Some 
technologies were discipline-specific whereas others were more generic. All classrooms were 
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outfitted with Smartboards, a computer for the teacher, a projection system, and classroom sets 
of laptops or tablets. The computers were stored in mobile containers called COWs or computers 
on wheels. WiFi was readily available throughout each campus, although one teacher noted that 
the bandwidth was insufficient for the demand. Other technologies included WiFi-enabled 
calculators, graphing calculators, student response systems, smart phones, laboratory equipment 
specific to the science courses, and electronic keyboards in the music rooms. Teachers used the 
Galileo formative assessment system, various applications provided by the Arizona Department 
of Education such as AZDash, videos, Apple TV, YouTube materials, Schoolology, notebook 
software, GradeMaster, Google Earth, Google Classroom, Kahoots, Garage Band, My Fitness 
PAC, PowerPoint, interactive journals, spreadsheets, and many other websites, software 
appropriate for their disciplines. Teachers also used text messaging, Facebook, and email to 
communicate with their students. 


Below are some examples of the use of the applications. 

¢ A music teacher used Garage Band for composition. She also used the electronic 
keyboards to record student performances and monitor progress from which she could 
diagnose issues and discuss them with students. This teacher also used videos to provide 
examples of musical expression. 

¢ An English teacher had her students access websites to explore advertising campaigns, 
analyze messaging, and write about the findings. 

¢ A science teacher used Google Earth to demonstrate aspects of ecology and earth science. 

¢ Another science teacher used videos of animals and plants for biology and then 
concretized the videos with school trips to the zoo and botanical gardens. 

¢ An English teacher used remote technologies to monitor students’ performance on 
writing assignments, especially when absences rose. 

¢ A business teacher used websites to have her students construct a marketing plan for a 
professional baseball team. She also used websites to simulate the job application 
process. 

¢ A social studies teacher used Google Earth to examine the boundaries of countries. 

¢ A math teacher used the Smartboard to collect student responses to a unit on 
measurement. He was able to examine individual student performance as well as the 
whole class to look for understanding and misconceptions. 

¢ Many other teachers used the Smartboards for the same purpose — recording and 
analyzing student work, in English, music, social studies, science, math, and business. 

¢ A physical education teacher used a fitness app to measure activity levels. She also used a 
nutrition app to help her students make better decisions about food choices. 

¢ A math teacher used the student response system to engage students in a discussion about 
a unit on equations. 

¢ Ascience teacher used social media to communicate with her students about a particular 
unit. 


All of the teachers relied on the affordances of Galileo to provide real-time assessments that are 
linked to state standards. They commented on the expedience of turnaround time of the analyses 
and reporting and the system’s deep item analyses and explicit graphical displays. They also 

commented that the system allowed teachers and students to monitor performance at a level that 
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provided invaluable information about student successes and failures, understandings, and 
misconceptions. 


The fact that each classroom had laptop computers or tablets in sufficient number for all students 
enabled teachers to use the Galileo system effectively as well as have students conduct research 
and online activities individually or in small groups as appropriate for the lesson. 


One of the most essential affordances of the technologies was the data that the applications 
provided to teachers and students. A primary source was from Galileo, providing rich and multi- 
level formative assessment data almost instantaneously (see Appendix B). Teachers 
acknowledged that it would have been almost impossible for them 


to generate these data without spending significant amounts of One of the most essential 
time. Galileo provided item analyses down to the level of affor dances of the 
distractor analyses so that teachers could immediately identify technologies was the 


data that the 
applications provided to 
teachers and students. 


student misconceptions. For teachers to do this without the 
technology would have been an onerous and labor intensive 
process. In addition to the real-time data and the immediacy of the 
analyses, Galileo also provided easily understandable tabulated 
results and reports aligned to state standards. Other teachers noted the diagnostic value to the 
data, not just from Galileo but other technologies. The Smartboards also enabled teachers to 
collect, record, and analyze student work products. 


One of the challenges to having so many technologies is the fact that they often work in isolation 
with limited interoperability. Indeed, this was the case across the three schools. Applications 
worked in isolation. Connections were rare and data sharing pretty much non-existent. 


Data Sources and Use 

Teachers were asked about their access to data and the sources of data they use. They all reported 
that they had access to many sources of data. They also noted that their schools had a “data guy” 
who was the go-to person for all things related to data. He would help them access data and help 
them with the data use process. Most of the teachers focused on the data from the Galileo 
assessment system, noting that without the system and its analytics, the data process would be 
impossible and too labor intensive. 


Yet when asked about the kinds of data they use, teachers were clear that they used much more 
than assessment and student performance data. Some teachers mentioned the Galileo data, some 
mentioned state assessment data, but others identified motivational, perception, attendance, 
behavioral, observational, attitudinal (mood), physical, medical, group processes, background 
(family), demographic, and coping strategy data. It was clear, though, that the data from Galileo, 
and the system’s ability to drill down to item analyses loomed large as a data source. Because of 
the frequent conflation between data literacy and assessment literacy (Mandinach & Gummer, 
2016), teachers were asked specifically if data were only about assessments or data are broader. 
All teachers acknowledged the centrality of assessment data but also the need to consider the 
whole child, context, and background. The teachers were asked what they thought were the most 
important data. Responses were diverse, including cognitive, non-cognitive, and contextual data 
(see Table 2). 
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Table 2. 
Reported Data as Important 


Assessment (formative Interests 

and others) 

Attendance history Living conditions 
Background data Observations 
Confidence Perception 
Context Progress over time 
Home life Trust 

IEP's Willingness to try 


Yet even with the plethora of data available to them, the teachers acknowledged that there were 
data they would like to have. Many wanted historical or longitudinal data. They wanted more 
data on family issues and background (graduation history), arrest records, and background or 
contextual data to help them gain a more holistic understanding of their students. One teacher 
even mentioned that she wanted photos of each student. Another teacher connected with students 
through Facebook. Students followed the teacher’s out of school activities and the teacher was 
able to learn a great deal about what is important in her students’ lives. She was able to better 
contextualize her work with them by understanding home and personal environments 


Teachers were asked if the data they currently have are sufficient for making decisions. They all 
said yes, but noted that there were other data that would be helpful. Some of the missing data 
were arrest records, data on family issues, emotional data, and data one teacher termed 
“intangible” (e.g., whether a student is having a bad day). All teachers reported that the data they 
have are actionable; that is, from those data, they can draw conclusions and make decisions. The 
data help them to determine students’ understanding and misconceptions and what needs to be 
retaught. 


Artifact Protocol 


The Artifact Protocol was given to all teachers during their initial interview. It asked the teachers 
to identify the data they use in the course of their practice, the materials or resources they 
typically access, and the colleagues and specialists they consult about their instruction. Results 
reflect data from 14 teachers. 


Teachers reported using diverse sources of data, with some being fairly universal. Assessment 


data are the most frequently used, followed by conversations with students, colleagues, and 
parents, classroom activities, observations, attendance, and behavior (see Table 3). 
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Table 3. 


Number of Teachers Reporting Use of Types of Data 
iWiek=melm Blejie) iN) iieX=ne)m Blejce) 


iN 
Summative data 14 Conversations with parents 10 
Formative data 13 Student reports 9 
Conversations with students 13 Other student work products 9 
Quizzes 12 Attitudinal data 6 
Class assignments 12 Medical data 6 
Observational data 12 Demographic data 6 
Attendance data 1] Interim data 4 
State achievement data 11 Other: 
Conversations with colleagues 1 PLC 1 
Behavior data im Student projects 1 
Benchmark data 1] 


Teachers reported that they use diverse materials and resources. These include research (10), 
public media (11), textbooks (9), and diagnostic materials (11). All but one teacher reported 
using a variety of websites. They consulted websites for their specific disciplines, assessment 
websites (such as the Arizona site AZDash and the formative assessment site, ATI-Online), 
social media, and professional websites. All respondents reported that they sought help from and 
consulted colleagues and other professionals. These included administrators, other teachers, the 
school “data guy”, supervisors, former professors, WestEd (provider of technical assistance and 
professional development), the formative assessment professionals at ATI, colleagues at 
conferences, disciplinary specialists, and state data specialists through AZDash. 


Classroom Observations 


Classroom observations were conducted for all participating teachers. Some teachers were 
observed multiple times. The observations noted the classroom set up, forms of technologies in 
the classroom, media and resources use, type of classroom activities, number, composition, and 
grouping of students, and structure of the classroom. 


All classes were small in terms of number of students. The largest class had 20 students, but 
most were 15 and under. Some classes even had 5 to 7 students. Given the demographics of the 
three schools, the student populations were diverse with a predominance of Hispanic and African 
American students. 


Technology loomed large in all classrooms. Every room was outfitted with a SmartBoard, 
projection system, computers, laptops, and tablets. Other technology applications were used as 
needed, depending upon the discipline (e.g., lab equipment for science courses, synthesizers for 
music classes). All classrooms contain high-bandwidth WiF1. 


Classrooms in the two high schools were set up around student groups to allow for maximum 
student interactions. Seating was in pods of students. These classrooms were centered around the 
use of the SmartBoards. In contrast, the classrooms in the middle school were set up more 
traditionally, in rows to accommodate the larger number of students. These classrooms also 
focused on the SmartBoards but also contained what were called “COWs” — computers on 
wheels. Each classroom had a large mobile container on wheels from which teachers would 
extract laptops or tablets to distribute to the students for personalized work. 
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Examples of Classroom Observations Using the Affordances of Blended Learning 
Classroom observations yielded rich examples of how different teachers took advantage of the 
technologies to enhance the teaching and learning process. These examples illustrate that impact 
of the affordances of the technology. 


The music teacher in one of the high schools used the SmartBoard to show video clips of 
music from popular movies to illustrate the concept of leitmotif in music. Students had to 
name the song and the movie and discuss the theme of the music and the emotions the 
music conveyed. Students then worked on keyboards (individually or in pairs), linked to 
a central panel for the teacher to hear their performances based on their understanding of 
the lesson. 

The music teacher, although not working in a tested area, participated in data teams for 
language arts and math. She included lessons on notes to link to the concept of fractions 
for the math team. She asked students to write about their favorite composer or artist for 
the language arts data team. 

A high school math teacher taught a lesson on seriation. Students sat in small groups with 
tablets to work on the lesson. The teacher used the SmartBoard where students presented 
their work and defended their responses to the problem. 

A high school math teacher taught a lesson that required students to build a catapult. She 
brought in tools to help the students gain hands-on experience with the construction 
process, focusing on the underlying math concepts. This was a revision of an original 
lesson on the same math topic where she had presented the topic first, examined the 
student performance data, and determined that the students were struggling with the 
math. She then modified the order of the lesson to do the hands-on component first to 
concretize the concepts first, then followed by the math. Students were self-monitoring 
their progress through the hands-on component and through online assessments of the 
math topic, aligned to the state standards. 

A student approached a high school math teacher with his mobile device in hand. He 
proudly announced to the teacher that he had worked through 80 percent of the online 
assessment remotely from home the night before. The teacher had set a deadline for 
completion of the unit and the student was remotely working through the assessments and 
working with the teacher on an individual basis to meet the personalized timeline for 
mastery of the materials. 

A high school science teacher mixed instructional approaches on a biology topic. She 
provided selected video clips for the class to view. She called individual students to her 
computer console to review their progress on the unit. During this review, she retrieved 
each students’ electronic portfolio of work to discuss what they had done well and topics 
that still needed attention. The teacher organized a sequence of class trips to local venues 
to strengthen their understanding of the biology content. These trips included the zoo and 
botanical gardens. 

A middle school social studies teacher conducted a lesson on income disparities in 
Middle Eastern countries. She provided a structured worksheet with questions for her 
students that helped to facilitate their online research on income. The students were 
required to access media from the specific countries to research topics such as average 
income and gender income disparity. Secondary sources were not allowed. The teacher 
then asked the class about their findings from their research, asked for the source of the 
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information, and engaged them in a discussion about income in the targeted countries in 
comparison to similar data from Arizona and the United States. 

A high school English teacher worked with her class on an exercise on advertising. 
Students were grouped together, using tablets, to research particular advertisements 
around which structured questions helped the students conduct their research. The groups 
conducted their research while the teacher moved from group to group, stimulating 
discussion about their findings. She then brought the class together for a full group 
discussion and presentation of group findings. 

A high school teacher discussed performance goals with a particular student. This student 
was particularly challenged because he has to take public transportation that requires two 
hours of commute time to get to school. This means he often misses his first period class 
and had to do the work remotely. The teacher worked with the student to ensure that he 
was on pace with the progression of work and that he has access to the materials and the 
assessments via the student’s mobile device. The teacher notes, that without the mobile 
device access, this student would not be able to complete the course and graduate. 

A middle school physical education teacher covered a unit on nutrition and fitness. She 
had her students access a website that laid out the key principles of good nutrition and 
fitness. She had them download an app to calculate their activity relative to the principles. 
She discussed with the class dichotomies such as taking the stairs versus the elevator, 
eating grilled versus fried chicken, and eating at home versus eating out. The students 
charted their eating and exercise patterns over time and relate this to the key principles. 
A high school career and technical education teacher had her class prepare for a job fair 
and conduct a job search in another city. The learning objective was how to apply for a 
job in a service industry in a distant city. Students were required to research the available 
jobs in the city, the needed qualifications, and the application process. This required them 
to go online to various websites and accumulate information. Students worked 
individually but discussed their findings with a partner as if the partner were an 
interviewer. The class then came back together as students showed the product of their 
search and a cover letter for a job application. 

A middle school social studies teacher gave an online formative assessment to his 
students using the Galileo assessment system. Students had either laptop computers or 
tablets on their desks to take the assessment. The teacher’s console showed students’ 
responses as they were taking the test. The console showed a matrix of student by item, 
displayed down to the level or correct or incorrect responses and what distractor students 
had selected. The teacher was able to examine the matrix to determine how well the 
students understood the particular concept, linked to the state standards. He could tell 
which students were struggling, which students had mastered the concept, and which 
specific topics are causing the more learner difficulties. The teacher then reported to the 
class what needed to be retaught to the entire class and what needed to be a focus for 
particular students. He was able to discuss immediately with each student how they had 
done and what the next instructional steps would be. When asked later about the use of 
Galileo, the teacher admitted that he might be able to do the same sort of analyses 
without the technology but the construction of the assessment, the immediate logging of 
responses, the analyses, and the reporting resulted in significant time savings. He 
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estimated it would have taken him several hours to do what Galileo had done 
instantaneously. 

¢ A high school career and technical education teacher worked with her students to develop 
and run the school store, based on principles from her business courses. This required 
students to understand the fundamental concept of supply and demand and use 
technology to monitor inventory and finance. Students, under the guidance of the teacher, 
collected data from the store, analyzed purchasing trends, determined pricing and 
restocking strategies, and ran the store as a successful technology-based enterprise. 


Because Galileo was such an important tool across the observations and in the interviews with 
the teachers, ATI, the developer of Galileo has provided sample screenshots to illustrate what the 
teachers see, from which they can extract information about student performance, determine 
subsequent instructional steps, and stimulate performance discussions with students. See 
Appendix B for the screenshots. 


The observations described above indicate that the affordances of the technologies in the blended 
learning environments were being used creatively and effectively to meet the individual needs of 
students and to customize instruction in ways to facilitate student learning. That said, there were 
observations where the classrooms looked no different than more traditional ones. Students sat in 
orderly rows. The teacher used a didactic, teacher-centered approach, and the use of technology 
was minimal. For the most part, however, the teachers in the three schools used the technologies 
available to them for planning, for instruction, and for assessment. Students used the mobile 
devices for remote learning and assessment, and for activities in the classroom. Teachers were 
able to monitor student performance and discuss results with the students to keep them on pace. 
Students were engaged in the performance conversations. Data were readily available to teachers 
and students that might have been impossible or impractical without the technologies. 


Data Literacy Protocol 


The data literacy protocol was developed to capture the skills, knowledge, and dispositions 
outlined by Mandinach and Gummer as they created the data literacy for teachers (DLFT) 
construct (Gummer & Mandinach, 2015; Mandinach & Gummer, 2016). The protocol was 
reflective of an intermediate version of the construct. Since the time of the developing and using 
the protocol, the construct has further evolved and skills and knowledge categorized in a 
modified manner (Mandinach & Gummer, 2016). The Protocol was aligned to the components of 
the DLFT construct with six skills related to the Identification of Problems of Classroom 
Practice, 21 skills related to Using Data, 14 to the Transformation of Data Into Information, nine 
to the Transformation of Information Into a Decision, seven to the Evaluation of the Outcomes of 
a Decision, and seven to Data-Related Habits of Mind. Given the construct, it was expected that 
some of these skills would be manifested in classroom practice and observable through 
classroom observations, whereas others might be implicit, exhibited during data teaming 
activities, or done outside of the classroom. Furthermore, whether or not the skills might be 
evident through observation may be dependent upon the classroom activities being observed at 
any given time. The results were expected to reflect those nuances. 


Appendix C provides the data from the protocol and illustrates the components original coding 
under which the data skills and knowledge were categorized. Data were collected for 23 
classroom observations, seven of which were repeated measures to determine consistency within 
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a teacher. Results from the Protocol are presented in Table 4. The first column contains a list of 
the specific skills. The next column denotes the number of teachers who showed evidence of the 
skill during the observation. The third column indicates how many among the seven repeated 
observations showed consistency, either in the presence or absence of the skill during the 
observation. 


As can be seen in Table 4, the range of observed skills is from 2 (determine unintended 
consequences) to 23 (monitor student progress). The most readily observable skills are included 
under the component, Transform Information into a Decision, the essential pedagogical 
component of DLFT. These are the skills that teachers use to impact instructional practice. The 
most frequently observed skills in this component include individualizing instruction, planning 
classroom practice, designing classroom practice, and implementing classroom practice (22 
teachers). Almost all the teachers also showed evidence of articulating a problem of practice 
(22), monitoring classroom practice shifts, (22), using both formative and summative 
assessments (21), adapting current classroom practice based on immediate student feedback (21), 
incorporating student work as a data source (20), recognizing and using informal classroom 
information as a source of data (20), adjusting (slight change in method) classroom practice (20), 
and engaging students with their personal results (20). 


The least frequent skills may be those that are not readily observable in classroom practice. They 
include: the ethical use of data (3); belief in data (4); comparing data pre- and post-decision (4); 
probing for causality (4); identifying inaccurate, misleading, or out-of-range data (4); prioritizing 
data (4); understanding basic statistics such as measures of central tendency and dispersion (5); 
using statistics (5); re-analyzing the original problem or decision (5); testing hypotheses (5); and 
considering the need for an iterative decision cycle (5). 


In the case of multiple observations, there was much coherence within teachers of either showing 
evidence of specific skills or not using those skills. For example, all seven teachers showed 
consistency across the observations on: understanding the contextual issues at the student level; 
understanding the purposes of different sources of data; understanding what data are not 
applicable to the problem; integrating data sources; incorporating student work as a data source; 
recognizing and using informal classroom information as a data source; generating hypotheses; 
determining the next instructional steps; planning classroom practice; designing classroom 
practice; implementing classroom practice; monitoring for student progress; monitoring 
classroom practice shifts; analyzing relevant student work for evidence of changes in thinking; 
engaging students with their personal results; and the ethical use of data. 


Table 4. 
Results from the Data Literacy Protocol 


DYoH KON) <II Garon iuisxelel= Teachers Observed Nl elsYelK=xe| 
Usiakemsy.dll Olosareniolas 


Articulate the problem 22 6 
Aware of student privacy 6 3 
Understand contextual issues — student level 19 7 
Understand contextual issues — school level 9 3 
Involve stakeholders 1] 6 
Frame research questions 14 4 
Understand applicable data 12 5 
Identify possible data sources 12 6 
Understand purposes of different data 1] 7 
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Understand what data are not applicable 10 7 
Access, retrieve data 1] 4 
Use multiple sources of data 16 5 
Use qualitative and quantitative data 2 5 
Understand data quality 6 5 
Identify inaccurate data 4 6 
Integrate data sources 9 7 
Manipulate data 6 5 
Examine data 1] 4 
Prioritize data 4 Ns) 
Organize data 6 w) 
Understand appropriate levels of data is 4 
Understand basic statistics 5) 4 
Recognize valid and reliable assessments 8 ) 
Use both formative and summative 21 6 
assessments 

Incorporate student work as data source 20 7 
Use technologies to support data use 19 4 
Diagnose/identify student learning needs 19 6 
Analyze data 16 5 
Test assumptions 14 3 
Generate hypotheses 16 Hf 
Predict possible or likely consequences 6 6 
Use statistics S) 5 
Understand data displays and 9 4 
representations 

Assess patterns and trends 8 6 
Synthesize diverse data 8 5 
Summarize data 6 3 
Communicate about data 1] 2 
Interpret, make meaning 13 4 
Draw inferences and conclusions 15 4 
Probe for causality 4 1 
Understand context for decision 15 S) 
Determine next steps 19 i 
Individualize instruction 2D) 6 
Differentiate classroom practice 18 5 
Adjust classroom practice 20 5S) 
Plan classroom practice DD i 
Design classroom practice 22 7 
Implement classroom practice 21 6 
Adjust current classroom practice 5S) 6 
Re-analyze original problem or decision 5) S) 
Test hypotheses 22. 7 
Monitor classroom practice shifts 22 i 
Monitor for student progress 23 d/ 
Analyze relevant student work 19 Hf 
Compare data pre- and post-decision 4 4 
Determine unintended consequences D 5 
Consider need for iterative decision cycle 5 5 
Engage students with personal results 20 7 
Demonstrate ethical use of data 3 I! 
Demonstrate belief in data use 4 5 
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Think critically i 3 
Collaborate 9 ] 
Communicate with multiple audiences 2 6 
Engage in continuous inquiry cycle 2 6 


Survey Results 


Teachers were asked to complete a survey during the first year of the data collection. The survey 
covered a number of topics that relate to school culture, technology, data, and leadership. The 
survey can be found in Appendix D. 


Teacher Background Information 


Fifteen teacher respondents completed the survey. Of the 15°, there are five English language 
arts, reading, and/or writing teachers; five mathematics teachers; two science teachers; two social 
studies teachers; one visual or performing arts teacher; one physical education teacher; one 
career/technical education teacher; and two teachers categorized themselves as teaching other 
subjects. Respondents teach grade levels ranging from 6" through 12" grade: three teach 6" 
grade, four teach 7", three teach 8", seven teach 9", nine teach 10", eight teach 11", and eight 
teach 12". Teaching experience ranges from one year to ten years or more. Seven teachers have 
three or fewer years of experience and eight teachers have more than three years of experience. 
Specifically, three teachers have one year of experience, two have two years, two have three 
years, one has four years, one has five years, one has six years, one has seven years, one has 
eight years, one has nine years, one has ten years, and one teacher has more than ten years of 
teaching experience. 


Concerning teacher certification, 11 teachers hold a regular or standard certificate to teach in the 
state while four do not hold such a certificate and are also not currently enrolled in a certification 
program. To enter the education field, eight teachers completed an undergraduate teacher 
preparation program, three teachers completed a graduate teacher preparation program, one 
completed an alternate teacher preparation program (e.g., Teach For America or a program 
aimed at mid-career changes), two completed courses after attaining an undergraduate degree but 
not as part of a formalized graduate or alternative teacher program, one has not completed a 
formal preparation program, and one described his/her preparation as being described in another 
way but did not provide a description. 


Teacher Views and Beliefs of Colleagues 


Teachers largely hold favorable views of their colleagues’ and administrators’ efforts to support 
one another and focus on student learning. The majority of respondents agree that teacher 
colleagues collaborate with one another (12 of 15 teachers) and are highly focused on the 
mission of improving student learning (13 of 15 teachers). The majority of respondents agree 
that administrators at the school are highly supportive of teachers (13 of 15 teachers), are highly 
focused on student learning (11 of 15 teachers), and trust teachers to make decisions about their 
own instruction (13 of 15 teachers). 


° Note that teachers may teach in more than one subject area or grade level so the numbers may total to more than 
15. 
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Teacher Views of School Conditions That Promote Student Learning Through 
Technology 


Teachers identified a select number of school conditions that serve as major obstacles to their 
efforts to promote student learning using technology. Nine teachers believe that students’ 
inadequate technology skills serve as a minor obstacle to student learning, while six teachers 
believe this condition is not an obstacle. The majority of respondents (13 teachers) believe any 
limitations they might have regarding technology skills do not present an obstacle to student 
learning, while two teachers believe their limitations are a minor obstacle. Six teachers believe 
the lack of support from technology specialists or other staff who provide technical support is an 
obstacle to student learning, while the majority of respondents (nine teachers) believe this lack of 
support is not an obstacle. 


Six teachers believe an inadequate number of computers and devices to accommodate all 
students is a problem within the school that affects student learning. Seven teachers believe a 
lack of devices is not an issue within the school and two teachers believe there to be a lack of 
devices for all students, but that this issue does not affect their learning. The majority of 
respondents (11 teachers) believe that a slow Internet connection and/or inadequate bandwidth is 
an obstacle within the school that effects student learning. Three other teachers believe the 
Internet to be slow and/or the bandwidth to be inadequate but do not believe this to be an 
obstacle to student learning. One teacher does not view the Internet connection and bandwidth to 
be an issue. The survey results conflict, to some degree, with the interviews and observations. 


Four teachers believe they have had inadequate opportunities to participate in professional 
development related to technology use, which has been a minor obstacle to their efforts to 
promote student learning. Three teachers also believe there have been inadequate opportunities 
for professional development but do not believe this to have been an obstacle to student learning. 
Eight teachers do not believe the professional development opportunities related to technology 
use available to them have been inadequate. These results also conflict with the information 
yielded in the interviews where teachers reported a multitude of professional learning 
opportunities. 


The majority of respondents (nine teachers) believe that problems with hardware have been 
obstacles to their efforts to promote student learning. Two teachers believe such problems exist 
but they do not hinder student learning. Four teachers do not believe there to be issues with 
hardware. 


Teacher Views of School Conditions That Promote Blended Learning for Students 


Generally, teachers have identified a select number of school conditions that serve as obstacles to 
their efforts to promote personalized learning for students. Five teachers believe a lack of support 
from school administration is an obstacle to their efforts to promote personalized learning for 
students. Four teachers believe there is a lack of support but do not view this to be an issue that 
hinders personalized learning for students. Six teachers do not believe a lack of support from 
administration is an issue within the school. These results are interesting because all interviews 
yielded reports of the supportiveness of school leaders around the implementation and use of 
technology but the survey may have identified shortcomings in the support for implementing 
personalized learning. 
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In terms of their own expertise, eight teachers believe they have a limited knowledge of how to 
effectively personalize instruction. For six teachers, this gap in knowledge serves as an obstacle 
to promoting personalized learning for students. Seven teachers believe they have adequate 
knowledge to effectively personalize learning. 


The survey asked about class size and student load. The majority of respondents, nine teachers, 
do not believe their student load serves as an obstacle to promoting personalized learning. Six 
teachers believe they have too many students for whom they are responsible. For four teachers, 
their student load serves as an obstacle to promoting personalized learning. 


Teachers report that differences in student performance levels within classes is problematic. 
Twelve teachers believe there is too much diversity in achievement levels among their students. 
For eight of these teachers, this variance serves as an obstacle to their efforts to promote 
personalized learning. Three teachers do not believe their students’ achievement levels to be too 
diverse. 


Accountability pressures seem to loom large for the three schools. The majority of respondents 
(11 teachers) believe they face pressure to cover specific material as a result of state or district 
standards and/or testing requirements. For nine of these teachers, this pressure is an obstacle to 
their efforts to promote personalized learning. Four teachers do not believe they face this 
pressure. 


The majority of respondents (11 teachers) believe they spend excessive amounts of time 
developing personalized content for students. However, only five of these teachers believe the 
time needed to develop such content is an obstacle to promoting personalized learning. Four 
teachers do not believe they spend an excessive amount of time developing personalized content. 
Seven teachers believe they have inadequate data as they personalize students’ instruction, and 
two of these teachers believe inadequate data is an obstacle to their efforts to personalize student 
learning. Eight teachers believe they have adequate data. Eleven teachers believe they have an 
inadequate amount of time to prepare personalized lessons for all students, with six of these 
teachers believing this inadequate amount of time to be an obstacle to their efforts to promote 
personalized learning. Eight teachers believe they have an adequate amount of time to prepare 
personalized lessons for all students. 


Clearly absenteeism is a major issue in the schools given the student population. All teachers 
believe high levels of student absenteeism to be an obstacle to their efforts to promote 
personalized learning. Twelve teachers believe student absenteeism to be a major obstacle and 
three teachers believe it to be a minor obstacle. In addition to absenteeism, discipline is also an 
issue. All teachers believe high levels of student disciplinary problems exist within the school. 
Twelve of these teachers believe that student disciplinary problems present obstacles to their 
efforts to promote personalized learning for students. These results concur with the findings from 
administrator and teacher interviews. 


Type and Frequency of Student Performance Information Received by Teachers 
Teachers receive student performance information at varying levels of frequency. Some teachers 
receive certain information at least weekly while other teachers receive performance information 
a few times a year. Teachers responded to how often they receive: scores on assessments in 
mathematics or language arts; scores on assessments in subjects other than mathematics or 
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language arts; student performance on specific concepts or skills; identification of specific 
students who need extra assistance; identification of specific students who have achieved 
mastery; and non-achievement outcomes (e.g., student behavior and attitudes). 


Five teachers receive scores on assessments in mathematics or language arts at least weekly, 
seven receive this information at least monthly, and three teachers receive it a few times per year. 
Two teachers receive scores on assessments in subjects other than mathematics or language arts 
weekly, one receives this information monthly, eight teachers receive this information a few 
times per year, two receive it once per year, and two teachers never receive these scores. One 
teacher receives information about student performance on specific concepts or skills at least 
daily, five teachers receive this information at least weekly, five teachers receive it at least 
monthly, and four teachers receive this information a few times per year. 


Six teachers receive notifications of the students who need extra assistance at least weekly, four 
teachers receive this information at least monthly, and five teachers receive it a few times per 
year. One teacher receives daily notifications of the students who have achieved mastery, five 
teachers receive this information at least weekly, three receive this information at least monthly, 
four teachers receive it a few times per year, and two teachers are never informed of the students 
who have achieved mastery. Four teachers receive weekly notifications of non-achievement 
outcomes, three teachers receive this information at least monthly, four teachers receive this 
information at least once per year, and four teachers never receive information of non- 
achievement outcomes. 


Teacher Use of Student Achievement and Mastery Data 


Student performance data of course are a major data source for all teachers, whether in 
traditional or blended environments. Generally, the Pima teachers use student achievement and 
mastery data for a variety of reasons. Ten teachers use student achievement and mastery data to 
tailor the pace of instruction to individual students’ needs, while three teachers do not use data in 
this manner, and two teachers do not believe data is used to tailor instructional pace at the 
school. Ten teachers use student achievement and mastery data to tailor the content of instruction 
to individual students’ needs, while two teachers do not use data for this purpose, and two 
teachers do not believe data are used to tailor instructional content at the school. Thirteen 
teachers use student achievement and mastery data to develop recommendations for tutoring or 
other educational support services for particular students, while one teacher does not use data for 
this purpose, and one teacher does not believe data is used to develop student support 
recommendations. 


Fourteen teachers use student achievement and mastery data to identify topics requiring more or 
less emphasis in their instruction, while one teacher does not use data in this manner. Thirteen 
teachers use student achievement and mastery data to reflect on and discus teaching and learning 
with their colleagues, while two teachers do not believe data is used in this way at the school. 
Thirteen teachers use student achievement and mastery data to reflect on and discuss learning 
with their students, while one teacher does not use data for this purpose, and one teacher does not 
believe data is used to reflect and discuss with students at the school. 


Teacher Use of Non-Achievement Data 


Not all data that teachers use are solely achievement-oriented. Many other data sources exist. In 
the 2014-15 school year, five teachers reported using data on non-achievement outcomes (e.g., 
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student behavior and attitudes), while ten teachers did not use this type of data. Of these five 
teachers using non-achievement outcomes: four use it to tailor the pave of instruction to 
individual students’ needs; three use it to tailor the content of instruction to individual students’ 
needs; all five use it to develop recommendations for tutoring or other support services for 
particular students; and three use it to reflect on and discuss learning with their students. 


Teacher Reflections on the School Data Systems 


Teacher beliefs and attitudes can be strong influences on their actual practice so the survey asked 
about their feelings toward the use of data systems. In general, teachers have favorable views of 
their school’s data system. Ten teachers believe they have access to high-quality assessment data 
that help them adapt the pace or content of instruction to meet students’ needs, while four 
teachers do not believe they have access to high-quality assessment data. Ten teachers believe 
the data system at the school provides real-time data that are actionable, while four teachers 
disagree. Ten teachers believe the school’s data system provides information at a level of detail 
that helps them inform their instruction, while four teachers disagree. Eight teachers believe they 
have plenty of data but need help in figuring out how to translate the data into actionable steps. It 
is important to note that this is not a finding unique to the Pima schools as most professional 
development does not make the link between data use and actual instructional modifications 
(Mandinach & Gummer 2012, 2013, 2016). 


Nine teachers believe it is easy to create custom assessments that evaluate what students are 
learning, while six teachers disagree. Nine teachers believe the school’s data system is easy to 
use while five teachers disagree. Ten teachers report they can use the school’s data system to 
easily produce reports they need, while four teachers are not able to easily use the system. Ten 
teachers believe the technology provides data that are not typically available without technology, 
while four teachers disagree. 


Teacher capacity to use data systems and data more generally always is an issue, regardless of 
location (Mandinach & Gummer, 2012, 2016). Twelve teachers believe they have the necessary 
skills and experience to use data to guide their instruction, while two teachers disagree. One 
teacher did not find the questions on the school’s data system applicable to his/her experience. 


Teacher Data Sources and Frequency of Use 


The survey asked about usage patterns and the sources of data that the teachers access. In 
general, data closer to the individual student and classroom level (e.g., attendance and personal 
data) are used more frequently by teachers than other forms of data (e.g., state data). Three 
teachers use state data at least weekly, two teachers use them monthly, and ten teachers use state 
data less than once a month. Four teachers use periodic data at least weekly, five teachers use 
them monthly, and six teachers use them less than once a month. Six teachers use local data at 
least weekly, one teacher uses them monthly, and eight teachers use local data less than once a 
month. Seven teachers use personal data at least weekly, three teachers use them monthly, and 
five teachers use the data less than once a month. Six teachers use behavior data at least weekly, 
four teachers use them monthly, and five teachers use behavior data less than once a month. Four 
teachers use medical data at least weekly and eleven teachers use them less than once a month. 
Nine teachers use attendance data at least weekly, four teachers use them monthly, and two 
teachers use them less than once a month. Eight teachers use attitude data at least weekly, two 
teachers use them monthly, and five teachers use attitude data less than once a month. Nine 
teachers use motivation data at least weekly and six teachers use them less than once a month. 
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Data Sources and Their Usefulness 

Data will not necessarily be used unless educators perceive them to be useful. Utility can be a 
facilitator and a lack of perceived utility a major impediment. Table 5 represents the extent to 
which teachers reported the utility of different sources of data. 


Table 5. 

Teachers Find Several Data Sources Useful 

Data Source | _ Yes (N) No (N) 
Attendance data 15 ] 
Attitude data 14 ] 
Behavior data 14 ] 
Medical data 14 ] 
Motivation data 14 ] 
Personal data 14 ] 
Local data 12 3 
Periodic data 2 3 
State data 12 8 


State Data and Use 


Data from state data systems often are seen as lacking local utility. The Arizona Department of 
Education has taken on a major system development effort to create data tools for the state’s 
educators to provide them with real-time data. During the course of this project, the Department 
began rolling out some of the tools, though the effort is in its infancy. In general, state data is not 
used frequently throughout the school year. Two teachers use state data weekly to identify 
instructional content to use in class, seven teachers do so a few times a year, and three teachers 
use state data in this way one or two times a year. Two teachers use state data weekly to tailor 
instruction to individual students’ needs, six teachers do so a few times a year, and four teachers 
use state data in this way one or two times a year. Two teachers use state data weekly to develop 
recommendations for additional instructional support, six teachers do so a few times a year and 
four teachers use state data in this way one or two times a year. 


Two teachers use state data weekly to form small groups of students for targeted instruction, one 
teachers does this monthly, six teachers do so a few times a year, and three teachers use state 
data in this way one or two times a year. Two teachers discuss state data weekly with a parent or 
guardian, four teachers do so a few times a year, and six teachers use state data in this way one or 
two times a year. Two teachers meet with a specialist (e.g., an instructional coach or data coach) 
about state data on a weekly basis, one teacher meets with a specialist monthly, five teachers 
meet with a specialist a few times a year, and four teachers do so one or two times a year. Two 
teachers meet with another teacher about state data on a weekly basis, four teachers meet with 
another teacher monthly, four teachers meet a few times a year, and two teachers do so one or 
two times a year. 


Local Data and Use 

Local data typically have more utility at the school and classroom level. Yet among the Pima 
schools, the survey indicated that generally, local data are not used frequently by teachers. Four 
teachers use local data to identify instructional content to use in class on a weekly basis, five 
teachers use it once or twice a month, and two teachers do so less than once a month. Five 
teachers use local data to tailor instruction to individual students’ needs on a weekly basis, four 
teachers use it once or twice a month, and two teachers do so less than once a month. Four 
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teachers use local data to develop recommendations for additional instructional support on a 
weekly basis, five teachers use it once or twice a month, and two teachers do so less than once a 
month. Three teachers use local data to form small groups of students for targeted instruction on 
a weekly basis, six use it once or twice a month, and two teachers do so less than once a month. 


Two teachers discuss local data with a parent or guardian on a weekly basis, two teachers do so 
once or twice per month, and seven teachers do so less than once a month. Four teachers discuss 
local data with a student on a weekly basis, four teachers have a discussion once or twice a 
month and three teachers do so less than once a month. Two teachers meet with a specialist about 
local data on a weekly basis, five teachers meet once or twice a month, and for teachers do so 
less than once a month. Five teachers meet with another teacher about local data on a weekly 
basis, four teachers meet once or twice a month, and two teachers do so less than once a month. 


Teachers’ Support in Use of State Achievement Data and Locally Developed 
Assessments 


Teachers largely agree that they are supported in using state achievement data and locally 
developed assessments. Thirteen teachers believe they are adequately supported in the effective 
use of these data, while two teachers disagree. Twelve teachers believe they are adequately 
prepared to use these data, while three teachers disagree. Thirteen teachers believe there is 
someone who answers their questions about using the data, while two disagree. Eleven teachers 
believe there is someone who helps them change their practice based on these data, while four 
teachers disagree. Thirteen teachers believe the district provides enough professional 
development about data use, while two teachers disagree. Twelve teachers believe the district’s 
professional development is useful for their learning about data use, while three teachers 
disagree. 


The role of building leadership in data use is a well-established finding in the literature 
(Hamilton et al., 2009). Leadership in the Pima schools confirms that finding. All fifteen teachers 
believe the principal or assistant principal(s) encourages data use as a tool to support effective 
teaching. Fourteen teachers believe the principal or assistant principal(s) create many 
opportunities for teachers to use data, while one teacher disagrees. Twelve teachers believe the 
principal or assistant principal(s) has made sure teachers have plenty of training for data use, 
while three teachers disagree. Fourteen teachers believe the principal or assistant principal(s) are 
a good example of an effective data user, while one teacher disagrees. All fifteen teachers 
believe the principal or assistant principal(s) discusses data with them. Fourteen teachers believe 
the principal or assistant principal(s) creates protected time for using data, while one teacher 
disagrees. 


School and District Provided Technology 

As the Institute of Education Sciences practice guide on data use notes as one of the five 
recommendations of practice, having a school-wide or district-wide data system is essential 
(Hamilton et al., 2009). Through the project’s observations, the Pima schools have substantial 
technology resources. Teachers largely agree that the programs, systems, and other technology 
provided by the district are useful. All fifteen teachers believe they have the proper technology to 
efficiently examine data. Fourteen teachers believe the computer systems in the district provide 
them with access to lots of data, while one teacher disagrees. Twelve teachers believe the 
computer systems for data use in the district are easy to use, while three teachers disagree. 
Twelve teachers believe the computer systems in the district allow them to examine various 
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types of data at once, while three teachers disagree. Thirteen teachers believe the computer 
systems in the district generate displays and reports that are useful, while two teachers disagree. 


Teacher Attitudes Toward Data Use 


As noted above, teachers’ attitudes of and proclivity toward data use are important to effective 
data use. In general, the Pima teachers are comfortable with their ability to use data. Twelve 
teachers believe they are good at using data to diagnose student learning needs, while three 
teachers disagree. Twelve teachers believe they are good at adjusting instruction based on data, 
while three teachers disagree. Thirteen teachers believe they are good at using data to plan 
lessons, while two teachers disagree. Twelve teachers believe they are good at using data to set 
student learning goals, while three teachers disagree. 


Characteristics of the data systems can have an impact of teacher attitudes. Of the two major 
systems used in the Pima schools, Galileo is termed a formative assessment system that provides 
teachers with immediate data on teacher-constructed assessments. AZDash is a data dashboard 
being developed by the Arizona Department of Education to provide diverse data to educators. In 
general, the Galileo data system is found to be more informative than AZDash: Eight teachers 
report that Galileo is very informative, six teachers find it somewhat informative, and one 
teacher does not find it very informative; Four teachers report AZDash is very informative, eight 
teachers find it somewhat informative, and three teachers do not find it very informative. One 
teacher uses his/her own assessments due to not receiving training in either Galileo or AZDash. 


Teacher Attitudes Toward Collaborative Teams 


Most professional development and theoretical frameworks around data use focus on a model of 
collaborative inquiry carried out by data teams or professional learning communities (Hamilton 
et al., 2009; Love et al., 2008; Mandinach & Gummer, 2012, 2013, 2016; Mandinach & Jackson, 
2012). In general, teachers hold positive attitudes toward their collaborative teams. Ten teachers 
believe members of their team trust one another, while five teachers disagree. Nine teachers 
believe it is okay to discuss feelings and worries with other members of the team, while six 
teachers disagree. Ten teachers believe members of the team respect those colleagues who are 
experts in their craft, while five teachers disagree. Thirteen teachers believe the principal or 
assistant principal(s) foster a trusting environment for discussing data in teams, while two 
teachers disagree. 


Available Technologies and Their Usefulness 


Generally, teachers report the available technologies in the schools to be useful in their 
classrooms. Eight teachers report the computers on wheels to be very useful, four teachers find 
them somewhat useful, and three teachers do not find them to be useful in the classroom. Five 
teachers report the tablets very useful, six teachers find them somewhat useful, and four teachers 
do not see them very useful in the classroom. Twelve teachers report that the Smartboards are 
very useful, while two teachers find them somewhat useful, and one teachers does not find them 
useful in the classroom. Twelve teachers report the Internet connectivity to be very useful and 
three teachers find it somewhat useful in the classroom. Six teachers report that the mobile 
devices (smart phones) are very useful in the classroom, four teachers find them to be somewhat 
useful, and five teachers do not find them useful in the classroom. 
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DiscussION 


It is clear from the multiple sources of results that teachers in the Pima Prevention Partnership 
schools are using the blended learning environments of richly infused technology and the 
resulting sources of data to reach a challenged population of students. This might not be done as 
effectively and efficiently if it were not for the affordances provided by the technologies used in 
the classrooms. Teachers and students have any time and any where access that allows them to 
extend and maximize learning opportunities. Teachers have in-depth item analyses linked to state 
standards to help them monitor student progress and plan instruction through their interactions 
with the Galileo system. Teachers report that the blended environment changes the structure and 
function of the classroom and the role of the student. Teachers also report that students are more 
active, engaged, motivated, and involved in their own learning. 


Recalling the attributes that Pane and colleagues (2015) outlined for blended learning, all 13 
components were observed in the three schools. The schools used personalized goals to structure 
the instructional plans for each student. The teachers and administrators used student data to 
inform instruction and discussed and involved the students in data-based discussions. The 
teachers across the two schools used a variety of data and resources to meet the needs of the 
students. The data were real-time, plentiful, and available at all times. The technologies used 
across the schools provided any time and any where access to outside of school learning 
opportunities. This meant that the schools provided out-of-school learning opportunities through 
their access to the technological platforms. The teachers created flexible and multiple paths for 
students through content materials by customizing the instruction to the individual needs of each 
student. Students worked at their own pace. They also provided individualized student support. 
The Galileo assessment system helped teachers to ensure that students progressed through the 
content based on a competency model. Galileo also provided on-demand assessments through 
which students were able to demonstrate competency of the domain, linked to the state standards, 
and provided teachers with real-time data that could immediately inform instructional planning. 
The teachers used different grouping strategies in a flexible way, aligned to the characteristics of 
particular units and lessons. The classrooms in the Pima schools provided /earning spaces that 
supported the technologies and the diversity of learning objectives. Teachers used the 
affordances of the technologies to provide flexible learning time and extended learning time that 
often extended beyond the traditional and typical school hours in order to meet the needs of 
students’ transportation schedules. Teachers stayed late to work with students and provided 
communication strategies after school hours to meet the needs of the students. Technology was 
made available to all students. In class, each student had a dedicated laptop or tablet. Finally, the 
two high schools in particular, sought to provide students with life skills and college and career 
preparedness. The Arizona Collegiate High School’s vision included as an explicit goal to 
prepare its students for college. The Pima Partnership High School focused on career-ready skills 
through courses offered by the career and technical education teacher, whose courses 
emphasized job readiness. 


Taking the RAND report’s 13 attributes at face value, the Pima Prevention Partnership schools 
exhibited all of the components identified as important to blended learning environments. It is 
important to note, though, that these schools have characteristics that may not be readily 
generalizable. First, they are infused with many different technologies and teachers are well 
trained in how to use these applications. Second, one of the technologies, the Galileo assessment 
system, provides a rich source of real-time, student performance data that help teachers make 
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rapid turnaround instructional decisions and modifications. Third, the leadership of the three 
schools have an explicit vision for student performance, using the technologies and data to 
undergird all teaching and learning activities as well as administrative decision making. These 
schools have extremely strong and dedicated leaders that help to provide a conducive and open 
context that promotes collaboration, respect, and trust among the educators. Fourth, there is no 
doubt that the educators in the schools recognize the importance of how the blended 
environments provide contexts that are sufficiently flexible and customizable to attempt to meet 
the many difficult challenges of their student population. The technologies provide the any time 
and any where extension beyond the brick and mortar of the physical school buildings. 


In reviewing the characteristics of the Pima schools, it is important to note that the findings also 
are aligned to the five recommendations of the Institute of Education Science practice guide on 
data use (Hamilton et al., 2009). The first recommendation pertains to the use of an ongoing 
inquiry cycle to inform instruction. Clearly, the staff of these schools are identifying problems of 
practice, collecting data, and making decisions based on those data to inform their practice. The 
second recommendation is to involve students in the examination of their own data. Students in 
the three schools are taking an active role in being their own data-driven decision makers. 
Teachers work with them to set expectations and learning goals, and use data to monitor their 
progress to those learning objectives. The third recommendation is to establish a vision for data 
use. As noted above, the three building leaders explicitly lead with data. Their visions are well 
articulated and modeled for all to hear and see. The fourth recommendation is to create a culture 
of data use in the school. This means providing the necessary supports and resources for the 
establishment of a data culture. The schools have created data teams. There are dedicated times 
for the teams to meet. There is a “data guy” that helps all three schools with data-related issues 
and questions. School leaders make possible the time and resources for professional 
development, technical assistance, and attendance at relevant conferences and meetings. Finally, 
the fifth recommendation is to have a data system. The CMO has a central data system, but also 
uses other siloed applications, such as Galileo, that serve as repositories for data. Similar to the 
findings in other case studies (e.g., Bernatek, Cohen, Hanlon, & Wilka, 2012a, 2012b), the data 
systems lack interoperability and data sharing capacity, leaving each system to function in 
isolation. This is a very real issue due to the proliferation of diverse sources of data. 


The current findings also reflect the work from the SRI study (Murphy et al., 2014) as they 
pertain to data use in blended environments. The SRI work noted the importance of data cultures. 
WestEd’s work is in agreement. Whereas SRI noted that some online data were underutilized 
due to the quality of the data dashboards, WestEd found a strong emphasis on using online data, 
particularly from Galileo. The dashboards were an impediment, whereas Galileo was user- 
friendly and easily accessible. Galileo linked the data to state standards and produced 
interpretable data displays. Whereas data from SRI’s dashboards did not provide a path to 
instructional steps, Galileo’s results did. SRI mentioned a lack of trust in the online assessments. 
The Pima teachers did not convey any such level of doubt or mistrust of Galileo’s data. Both the 
SRI and WestEd findings indicated that data sharing and interoperability of data systems are 
challenges. 


Moreover, another challenge looms large which this work did not address but must be 
mentioned. As noted in this study, blended and personalized learning environments have the 
capacity to generate a plethora of data that can be accessible to teachers, students, and 
administrators. With the proliferation of data from diverse sources come concerns about the 
protection of data privacy. The collection and examination of these data sources, particularly in 
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real time, provide for great opportunities for the customization of teaching and learning. Yet 
even greater challenges exist in terms of maintaining the safety and security of the data to protect 
students’ privacy and confidentiality (Data Quality Campaign, 2016; Herold, 2014) and ensure 
that the educators are knowledgeable about how to use data not just effectively, but responsibly 
(Mandinach & Gummer, 2016). Data privacy and security have become key issues and must 
continue to be an area of focus. 


Blended learning certainly has taken hold in the three Pima Prevention Partnership schools. 
These are small charter schools that serve a relatively small number of students. The question of 
the model’s generalizability is of concern. More traditional schools are not likely to have the 
resources found in the Pima schools. They may have less technology. They may lack a vision for 
the use of technology and data. They may not have an embedded culture of data use and 
technology that extends within and beyond the school boundaries. They may not be provided 
with the extensive learning opportunities found in the Pima schools in terms of teacher 
professional development, technical assistance, and conference attendance. That said, there is 
much that can be learned from these three schools about how the alignment of RAND’s 13 
attributes (Pane et al., 2015) and the practice guide’s five recommendations (Hamilton et al., 
2009) in school settings can create blended or personalized learning environments that have the 
potential to reach even the most challenged students and help them to succeed. 
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Appendix A 
Interview Protocol 
Teacher 
School 
Items for Semi-Structured Interview Protocol 
Blended Learning or Personalized Learning Environments 
Can you describe how your classroom differs from one that might be considered more 
traditional? 


What value-added has technology given in your classroom/school? 


Does the blended learning environment provide data that might not be available in a more 
traditional setting? 


What is the role of the student in this environment? 


Do you use a model of collaborative inquiry in your classroom/school; that is; do you work with 
other teachers to examine data? 


Does your school have a vision for data use? For the use of the technologies? 


Does school leadership make it clear that using data is important? Using the technologies? 


Technology 
Can you please describe the technology that you have in your school/classroom? 


Please elaborate on the data that the technology generate, provide access to, enable you to use? 


Are the technologies connected in some way that there is data sharing across platforms? 


Do these technologies provide you access to data that would not necessarily be available in a 
more traditional educational setting? 


Data Sources 
Can you please provide a description of the various forms of data you use in the classroom and 
with the technology to inform yourself about your students? 


Are the data only about assessments, or are there other sources of data? If so, what would they 
be? 
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What data do you think are most important to provide information about your students? 


Are there data that you would like to have but currently are not available to you now? What are 
they and how much you obtain them? 


Are the data you do have sufficient to make decisions about your students’ learning 
Are your data actionable; that is, do they allow you to figure out what instructional steps are 
needed for particular students? 


Your Background with Data and Beliefs 
How long have you been teaching? 


Did you receive training on the use of the technologies in the blended learning environment? 
Have you received training in how to use data? If so, please describe. 
Have you ever taken a course in data-driven decision making? 


Can you describe what it means for a teacher to be data literate? What are the skills needed to 
use data? 


Do you consider yourself to be data literate? If not, what do you need to come data literate? 


Would your school provide you with learning opportunities for continuing education, graduate 
education, professional development? 


Do you think that using data helps you to improve your classroom practices and your ability to 
structure instruction based on your students’ needs? 


Does your school make is clear that data are an important tool? Technologies? 


How important do you think data are for your teaching practices? 
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Appendix B 
Screenshots from Galileo® 


6 Thank you to ATI, Jason Feld, and Nancy Auslander for providing the project with these example graphics. 
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Appendix C 
Data Literacy Protocol 
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2015 Teacher Data Use Survey 


Contact Information 


Please verify the following information: 


*First Name: 


*Last Name: 


*Email Address: 


All fields with an asterisk (*) are required. 


*1, What subject area(s) are you teaching (or supervising) this year (2014-2015)? (*Required) 


Select all that apply. 


English/language arts/reading/writing 


Mathematics 


Science 


Social Studies 


Foreign language(s) 


Visual or performing arts (art, music, etc.) 


Physical education/health education 


Career/technical education 


OOO /OyO;O;O;O)/O 


Other subject area(s): 


ee —=-—_ x._—_————— Oz, 


*2. Please indicate the grade levels of the students you teach. (*Required) 


Select all that apply. 


Kindergarten 


a 


5 


5 


5 


oO}; WN|/O)a}A}wl]rm}a 
3 


OyOyOyOyOOO|O|O;O;O;OyO Oo 


Our school doesn't use grade levels 
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*3. Including this school year (2014-2015), how many total years have you been teaching, regardless of location? (Required) 


years 


*4. Do you hold a regular or standard certificate to teach in this state? (*Required) 


Select all that apply. 
C1 | Yes 
TC | No, but | am currently enrolled in a certification program. 


O 


No, and | am not currently enrolled in a certification program. 


*5. Through which of the following types of programs did you enter teaching? (*Required) 


Select all that apply. 


O 


Through an undergraduate teacher preparation program. 


Through a graduate teacher preparation program. 


Through an alternative teacher preparation program (for example, Teach For America, or a program aimed at 
mid-career changers). 


C 
i) 
i) 


Courses taken after attainment of undergraduate/bachelor's degree, but not as part of a formalized graduate or 
alternative teacher preparation program (for example, non-degree oriented courses needed for certification 
taken at a local or online university. 


No formal preparation program. 


OO 


Other, please describe: 


Ld 


*6. Rate your level of agreement with each of the following statements about your school. (*Required) 


Select one per row. 


Strongly , Strongly 
Disagree Eisele ag Agree 
*Teachers at my school collaborate with one another. O O O O 
*Teachers at my school are highly focused on the mission of 
improving student learning. O O O O 
*Administrators at my school are highly supportive of 
teachers. 0 O 0 O 
*Administrators at my school are highly focused on student 
learning. O O O O 
*Administrators at my school trust teachers to make O O O O 
decisions about their own instruction. 
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*7. Please indicate whether the following conditions exist in your school and the degree to which each is an obstacle to your 


efforts to promote student learning using technology such as computers, smartphones, or tablets. If the condition does not exist in 


your school, please mark "Not applicable." (Required) 


Select one per row. 


Not applicable; Condition exists | Condition exists | Condition is a 
condition does not but is not an and is a minor major 
exist in my school obstacle obstacle obstacle 

*Inadequate technology skills 
among students. O O O O 
*My own limited technology skills. O O O O 


*Lack of support from technology 
specialists or other staff who can O O O O 
provide technical support. 


*An inadequate number of 
computers or devices to O O O O 
accommodate all students. 


*Slow internet connection or 


inadequate bandwidth. O O O O 
*Inadequate opportunities to 
participate in professional O O O O 


development related to technology 
use. 


*Problems with hardware, such as 
insufficient computing power or O O O O 
lack of compatibility with software. 
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*8. Please indicate the extent to which each of the following conditions is an obstacle to your efforts to promote personalized 
learning for students. If the condition does not exist in your school, please mark "Not applicable." (*Required) 


Select one per row. 


Not applicable; 
condition does not 


Condition exists 
but is not an 


Condition exists 
and is a minor 


Condition exists 
and is a major 


disciplinary problems. 


exist in my school obstacle obstacle obstacle 
*Lack of support from school 
administration. O O O O 
*My own limited knowledge of 
how to effectively personalize O O O O 
instruction. 
*Too many students for whom | 
am responsible. O O O O 
*Too much diversity in 
achievement levels among my O O O O 
students. 
*Pressure to cover specific 
material as a result of state or 
district standards or testing O O O O 
requirements. 
*Excessive amounts of time | 
need to spend developing O O O O 
personalized content. 
*Inadequate data to help me 
personalize students’ instruction. O O O O 
*An inadequate amount of time to 
prepare personalized lessons for O O O O 
all students. 
*High levels of student 
absenteeism. O O O O 
*High levels of student 
anes : Oo O O O 
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*9. In general, how frequently do you receive the following types of information about the performance of your students? 


(*“Required) 


Select one per row. 


Never 


Approximately 
monthly 


Approximately 
weekly 


*Scores on 
assessments 
in 
mathematics 
or language 
arts 


*Scores on 
assessments 
in subjects 
other than 
mathematics 
or language 
arts 


*Information 
about student 
performance 
on specific 
concepts or 
skills 


*|dentification 
of specific 
students who 
need extra 
assistance 


*|dentification 
of specific 
students who 
have 
achieved 
mastery 


*Non- 
achievement 
outcomes (for 
example, 
student 
behavior, 
attitudes, or 
motivation) 
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*10. This year, to what extent have you used student achievement/mastery data for each of the following purposes? (Consider 
data provided by instructional software, interim assessments or quizzes, unit or end of course tests, state accountability tests, 
district benchmark/interim tests, & other tests). (*Required) (Select one per row.) 


My school Did not Used data | Used datatoa | Used data 
doesnitdo | use data to a small moderate to a large 
this for this extent extent extent 
*Tailoring the pace of instruction to O O O O O 
individual students' needs. 
*Tailoring the content of instruction to 
individual students' needs. 0 O O O O 
*Developing recommendations for 
tutoring or other educational support O O O O O 
services for particular students. 
*Identifying topics requiring more or less 
emphasis in instruction. O O O O O 
*Reflecting on and discussing teaching 
and learning with other teachers. O O O O O 
*Reflection on and discussing learning 
with my students. O O O O O 


*11. This year, have you used data on non-achievement outcomes (for example, student behavior, attitudes, or motivation)? 


(*Required) 


Select all that apply. 
| Yes 
| No 


*12. This year, to what extent have you used data on non-achievement outcomes (for example, student behavior, attitudes, or 
motivation) for each of the following purposes? (*Required) 


Select one per row. 


My school panpiod Used data Used data toa Used data 
doesn't do : to a small moderate to a large 
; for this at 
this all extent extent extent 
*Tailoring the pace of instruction to 
individual students' needs. O O O O O 
*Tailoring the content of instruction to 
individual students' needs. O O O O O 
*Developing recommendations for 
tutoring or other support services for O O O O O 
particular students. 
*Reflecting on and discussing 
learning with my students. O O O O O 
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*13,. Please indicate your level of agreement with each of the following statements(*Required) 


Select one per row. 


Strongly 
Disagree 


Disagree | Agree 


Strongly 
Agree 


Not 


Applicable 


*| have access to high-quality assessment data that 
help me adapt the pace or content of instruction to 
meet students’ needs. 


O 


O O 


O 


O 


*The data system provides real-time data that is 
actionable. 


*Our school's data system provides information at a 
level of detail that helps me inform my instruction 
(e.g. breakdowns for specific skills or topics). 


*| have plenty of data but need help in figuring out 
how to translate the data int instructional steps. 


*It is easy to create custom assessments that 
evaluate what students are learning. 


*Our school's data system is easy to use. 


*| can use the school's data system to easily 
produce the views or reports | need. 


*The technology provides data that are not typically 
available without that technology. 


*| have the necessary skills and experience to use 
data to guide my instruction. 


*14, On average throughout the school year, how many hours of instructional time do students experience per week in the 


class(es) you teach? (Required) 


hours 


*15. Teachers use all kinds of information (i.e. data) to help plan for instruction that meets student learning needs. How 
frequently do you use the following forms of data? (*Required) 


Select one per row. 


Less than once a Once or twice a Weekly or almost A few times a 
month month weekly week 
*State Data O O O O 
*Periodic Data O O O O 
*Local Data O O O O 
a O O O O 
Other O O O O 
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15.1 If you marked the "Other" option above, please specify the form of data here: 


*16. How useful are the following forms of data to your practice? (*Required) 


Select one per row. 


Not useful Somewhat useful Useful Very useful 
*State Data O O O O 
*Periodic Data O O O O 
*Local Data O O O O 
*Personal Data O O oO O 
Other O O O O 


16.1 If you marked the "Other" option above, please specify the form of data here: 


17. These questions ask about state data. In a typical school year, how often do you do the following? (If you indicated that state 
data is not available to you, please skip this question.) 


Select one per row. 


One or two times | A few times a Monthly | Weekly 
a year year 

Use state data to identify instructional content to use in 
pe O O O O 

Use state data to tailor instruction to individual students' 
needs O O O O 

Use state data to develop recommendations for 
additional instructional support. O O O O 
Use state data to form small groups of students for 

targeted instruction. O O O O 
Discuss state data with a parent or guardian. O O O O 
Discuss state data with a student. O O O O 

Meet with a specialist (e.g., instructional coach or data 
coach) about state data. O O O O 
Meet with another teacher about state data. O O O O 
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18. These questions ask about Local Data developed and used in your school or district. In a typical month, how often do you do 
the following? (If you indicated that Local Data is not available to you, please go to the next question.) 


Select one per row. 


Less than Once or twice Weekly or A few times 
once a month a month almost weekly a week 

Use local data to identify instructional content 
to use in class. O O O O 

Use local data to tailor instruction to individual 
students’ needs. O O O O 

Use local data to develop recommendations 
for additional instructional support. O O O O 
Use local data to form small groups of 

students for targeted instruction. O O O O 
Discuss local data with a parent or guardian. O O O O 
Discuss local data with a student. O O O O 

Meet with a specialist (e.g., instructional coach 
or data coach) about local data. O O O O 
Meet with another teacher about local data. O O O O 
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*19. These questions ask about supports for using data. For "data," please consider only state achievement tests and locally 


developed assessments. Please indicate how much you agree or disagree with the following statements. (Required) 


Select one per row. 


about data use. 


Strongly : Strongly 
Disagree Disagree | Agree Agree 
*| am adequately supported in the effective use of data. O O O O 
*| am adequately prepared to use data. O O O O 
*There is someone who answers my questions about using 
data. O O O O 
*There is someone who helps me change my practice (e.g., 
my teaching) based on data. O O O O 
*My district provides enough professional development about 
data use. O O O O 
*My district's professional development is useful for learning O O O O 


*20. These questions ask how your principal and assistant principal(s) support you in using data. They won't see your responses. 
For "data," please consider only state achievement tests and locally developed assessments. Please indicate how much you agree 


or disagree with the following statements. (*“Required) 


Select one per row. 


effective data user. 


Strongly ; Strongly 
Disagree Disagree | Agree Agree 
*My principal or assistant principal(s) encourages data use as O O O O 
a tool to support effective teaching. 
*My principal or assistant principal(s) creates many O O O O 
opportunities for teachers to use data. 
*My principal or assistant principal(s) has made sure teachers O O O O 
have plenty of training for data use. 
My principal or assistant principal(s) is a good example of an O O O O 


*My principal or assistant principal(s) discusses data with me. 


*My principal or assistant principal(s) creates protected time 
for using data. 
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*21. Your school or district gives you programs, systems, and other technology to help you access and use student data. The 
following questions ask about these computer systems. Please indicate how much you agree with the following statements. 


(*Required) 


Select one per row. 


Strongly ; Strongly 
Disagree Disagree | Agree Agree 
*| have the proper technology to efficiently examine data. O O O O 
*The computer systems in my district provide me access to lots 
of data. O O O O 
*The computer systems (for data use) in my district are easy to 
ties O O O O 
*The computer systems in my district allow me to examine 
various types of data at once (e.g., attendance, achievement, O O O O 
demographics). 
*The computer systems in my district generate displays (e.g., O O O O 
reports, graphs, tables) that are useful to me. 


*22. These questions ask about your attitudes toward your own use of data. Please consider only state achievement tests and 
locally developed assessments. Please indicate how much you agree or disagree with the following statements. (*Required) 


Select one per row. 


. heh Disagree | Agree en y 
lam good at using a a student learning O O O O 
*l am good at adjusting instruction based on data. O O O O 
*l am good at using data to plan lessons. O O O O 
*| am good at using data to set student learning goals. O O O O 
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23. As you think about your collaborative team(s), please indicate how much you agree or disagree with the following 


statements. If you do not have scheduled meetings to work in collaborative teams, please go to the next question. 


Select one per row. 


environment for discussing data in teams. 


Strongly ; Strongly 
Disagree issgree | Agree Agree 
Members of my team trust each other. O O O O 
It's ok to discuss feelings and worries with other members of 
my team. O O O O 
Members of my team respect those colleagues who are 

experts in their craft. O O O O 
My principal or assistant principal(s) fosters a trusting O O O O 


24. What else would you like to share with us about data use? 
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